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Module Objectives 

In order to demonstrate an understanding of the virtual-file-system interface and of OSF/1 's implementations of 
the S5, UFS, and NFS file systems, the student should be able to: 

• explain the use of the reference count in the system file table entries 

• explain the roles of the yfsops and vnodeops data stnictures and the abstraction of the file system concept 

• describe how the buffer cache has been parallelized 

• describe how directories are protected from concurrent updates 

• give the size constraints on files in the S5 and UFS file systems 

• explain how two threads may simultaneously extend the size of two different files within the same UFS file 
system 

• explain why it is necessary for a NFS server to maintain a queue of recent NFS requests 
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5-1. The Big Picture 
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Student Notes: File Systems 

The file subsystem is part of the UNIX portion of OSF/1 . The user interface to the file subsystem is that of UNIX. 
The implementation is primarily based on that of 4.4BSD. What has been added in OSF/1 is the parallelization of 
the file system. 

The VPS implementation is fi*om BSD but has been parallelized. The S5 implementation is fi-om SVR3 and has 
not been parallelized; it is included mainly for compatibility purposes. The UPS implementation is, of course, 
fi"om BSD and has been parallelized. The NFS implementation was originally done at the University of Guelph in 
Canada. It was modified by Bericeley and has been parallelized. 

Some of the material of this module is discussed in chapter 11 of Open Software Foundation, 1990a. 
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5-2. Representing an Open File 



Representing an Open File, part 1 
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Student Notes: Representing an Open File, part 1 

The set of open files is a property of the process as a whole. Thus, while in traditional UNIX the file descriptor 
table appears in the user stracture, in OSF/1 it i^pears in the utask structure. This structure is used to map file 
descriptors representing open files to system file table entries. Each system file table entry represents an open file. 
As discussed later, each active file (i.e., a file that is open or otherwise being used) is represented by a vnode that 
is entered in the active vnode table. Files are accessed via a kernel-supported buffer cache and the file itself is, of 
course, kept on disk. 
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5-3. Representing an Open File 



Representing an Open File, part 2 
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Student Notes: Representing an Open File, part 2 

In this picture we illustrate what happens when a file is opened. 

The lowest-numbered available file descriptor is allocated from the file descriptor table. Next, an entry in the 
system file table is allocated, and the file descriptor table entry is set to point to the system file table entry. A 
vnode for the file is allocated (or found if it already exists) and the system file table entry is set to point to it. 
Additional fields of the system file table entry are initialized, including: 

• a reference count 

• the allowed access (i.e., how the file was opened — ^read-only, read-write) 

• the offset (i.e., the location within the file at which the next transfer will start) 

In a multithreaded environment like OSF/1 's, the reference cou nt takes o n particular importance . A reference 
count of 0, of course, means that the entry is no longer being used. Race conditions, for instance one thread 
closing the file while another thread within the same task accesses the file, must be guarded against. In addition, 
data structures such as the system file table entry must not be deallocated while they are in use. 

To avoid these problems, when the file is op en the reference count is set to(2J[l for the file descriptor table entry 
and 1 for the thread performing the open system call). When the thread returns fi-om the call, it removes its 
reference, reducing the reference count to 1. 

If two threads of the same task concurrently close and write the file, the reference count first goes firom 1 to 2 (1 
for the file descriptor table entry and 1 for the thread within the write system caU; the file table's reference count 
is incremented by 1 at the beginning of each I/O system call, except for the close system caU). If the close 
system call completes first, the reference count wiU be reduced by 1, to eliminate the file descriptor table entry's 
reference. But there stiU is a reference corresponding to the thread performing the write system call, so the file 
table entry remains allocated and the file remains open until this thread returns from the call. Thus the reference 
count enables the kernel to ensure that the file table entry and file exist as long as a thread is using them. / 

Another race condition that must be dealt with concerns the individual file descriptor table entries: when a file is 
being opened, we need to ensure that the file is not accessed by any other thread until the open has completed. To 
accomplish this, the file descriptor table entry is not made to point to the allocated file table entry until the open 
completes. However, we must make certain that this file descriptor table entry is not allocated by some other 
thread. Thus, when the file descriptor table entry is allocated it is marked as reserved. Only when the open 
completes is it set to point to the file table entry. 
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5-4. Representing an Open File 



Representing an Open File, part 3 
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Student Notes: Student Notes: Representing an Open File, part 3 

Dup was invented to deal with the following problem. By convention, file descriptors 1 and 2 arc used for 
processes' nonnal and diagnostic output. Nomially they both refer to the display, and thus diagnostic output is 
intenningled with nonnal output. Suppose, however, one wanted to redirect both file descriptors so that all 
output, normal and diagnostic, was sent to a file. One might open this file twice, once as file descriptor 1 and 
again as file descriptor 2, thereby creating two system file table entries. As file descriptor 1 receives output, the 
offset field of its file table entry advances with each write. After 1000 bytes have been written (sequentially), the 
offset field is set to 1000, representing the current end-of-file. 

If at this point a diagnostic message is written to file descriptor 2, it wiU start at the beginning of the file, 
overwriting the data already there, since file descriptor 2's file table entry's offset is still at 0. This outcome is 
certainly not desirable. 

To solve this problem, the dup system call makes file descriptors 1 and 2 both refer to the same file table entry 
and hence share the offset. 
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5-5. Representing an Open File 



Representing an Open File, part 4 
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Student Notes: Representing an Open File, part 4 

In this slide we see the effect of two opens of the same file within the same task. 
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5-6. Virtual File Systems 



Generalizing the File System Concept 
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Student Notes: Generalizing the File System Concept 

In the beginning, UNIX supported only one type of file system. Modem UNIX systems now support multiple file 
system types. To represent different file system types, generalizations of the standard file system data structures 
are used. The scheme adopted in OSF/1 is based on Sun's virtual file system (VFS) technology (though the code 
has been entirely rewritten — ^it is adapted from 4.4BSD). 
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5-7. Virtual File Systems 
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student Notes: Virtual File Systems (VFS) 

VFS is the abstraction of a file system that provides a common interface to many different file systems. OSF/1 
cunently supports the local UNIX file systems (S5 and UFS) and a rcimplementation of Sun's NFS. 

Each instance of a file system is represented by a mount structure. The interface to the file system is represented 
by an array of entry points, the yfsops array, that is attached to the mount structure and defines operations on the 
file system as a whole. 
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5-8. Virtual File Systems 
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Student Notes: Vnodes 

Vnodes are the abstractions of files. They represent individual files; they contain generic information about files 
and refer to the file-system-specific infoimation on files (to inodes for UNIX files and to nfsnodes for NFS files). 
They also provide access to the various operations on the files — each vnode refers to an array of entry points 
called vnodeops. 
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5-9. Virtual File Systems 



Mounting File Systems, part 1 
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Student Notes: Mounting File Systems, part 1 

To place a file system in the tree structure directory hierarchy, one must mount it. A file system as a whole is a 
device that is named as a special file in the /dev directory. In order that the contents of this device be treated as 
files, they must be made to appear in the directory hierarchy. 
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5-10. Virtual File Systems 



Mounting File Systems, part 2 
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Student Notes: Mounting File Systems, part 2 

The contents of the file system are placed in the directory hierarchy when one issues the mount command. The 
mount command superimposes the root directory of the file system on top of the directory given in the mount 
command. Any attempt to follow a path to this directory leads one instead to the root directory of the file system. 
Thus the prior contents of the mounted-upon directory become invisible. 
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5-11. Virtual File Systems 



File System Data Structures, part 1 
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Student Notes: File System Data Structures, part 1 

The data structures in this picture show a single mounted file system, the root file system, which happens to be a 
UFS file system. The field rootfs points to the mount structure of the root file system. The mount structure 
points to a file-system-specific mount stracturc, in this case the UFS mount stnicture. Each active file within this 
file system is represented by a vnode that in turn points to the mount structure. Attached to the vnode is a 
file-system-specific per-file data structure, in this case the inode. The inode represents the file within the UFS file 
system and is stored permanently on disk. 
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5-12. Virtual File Systems 



File System Data Structures, part 2 
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Student Notes: File System Data Structures, part 2 

Here we see the effect of mounting an NFS file system in the root file system of the previous picmrc. The mount 
stmcture for this file system is linked to the mount structure of the root file system. The mounted file system's 
mount stmcture also points to the file-system-specific mount stmcmre, in this case the NFS mount stmcture. The 
vnode of the mounted-upon directory is set to point to the mounted file system's mount stmcture to represent 
where the file system has been mounted. This moimt stmcture in tum points back to the vnode. Attached to the 
vnodes of the active files of the mounted file system are nfsnode data stmcmres, which represent the remote files. 
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5-13. Virtual File Systems 



Open and Create: Flow of Control 
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Student Notes: Open and Create: Flow of Control 

Copen is called in the kernel in response to create and open system calls. As shown in the picture, copen calls 
falloc, which tfien calls ufalloc. On return from these, i.e., after the open file data structures have been set up, 
copen calls vm_ppen to initiate locating the file in the directory hierarchy. vn_open calls nameU which for each 
directory in the path calls the lookup routine associated with the directory's file system. The boxes with heavy 
outlines represent indirect references to a routine via a vector such as vnodeops. In particular, VOP_LOOKUP 
means to call the lookup routine listed in the vnodeops array attached to the vnode. 
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5-14. Virtual File Systems 



Reading and Writing: Flow of Control 
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Student Notes: Reading and Writing: Flow of Control 

Most of the work peifoimed in reading and writing a file occurs within the file system. However, some of the 
work occurs at the file-system-independent level. The first step in any I/O request is to copy the parameters of the 
I/O request into the uio structure (see the next slide). The next step is to find the file table entry and the vnode 
and to verify that the user has peimission to perform the desired operation 

An important next step for regular files and directories is to lock the file offset in the file table entry (using a 
blocking read-write lock) so as to make the operation ^mic. This is done to avoid a race condition in which two 
threads that share the same file table entry concurrently access the file. ( Locking ha sJbgen done incorrectij^ 
some versions of UNIX.) 
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5-15. Virtual File Systems 



The 




tructure 



lovec 



UaU«s*mdaJ_aa__>*aWWa 



no.ofl)«fl^ 



M^^m 






baiter 






struct uio 



start address. 



length 



start address- 



length 



start address- 



length 



struct i^^ 



buffer segment 1 /J 



JJJ/JJ//JJ/J^ 

buffer segment 2 /; 

f .f r ^ J' jr .■■'• ,/• ^ r ./ .,■'• y' ' 



'"yy"7yy"yyy"y././y;/W 

\ buffer segment n X/X' 

if X .i"^ .1^ .1^ .f if ,^ if .^ X / .y./ ,ii' 



5-15. 



© 1990, 1991 Open Software Pomidition 



5-32 



I 



Module 5 — File Systems 



Student Notes: The Uio Structure 

The uio structure represents a logical I/O request. Its contents represent what needs to be done to complete an I/O 
request; these contents are updated as the I/O request progresses through the system. The buffer may, in general, 
be composed of multiple segments, and hence an array of iovec structures is needed to refer to each of the pieces 
of the buffer. This organization is made necessary by the readv and writev system calls, which use such 
multicomponent buffers. 
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5-16. The Buffer Cache 
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student Notes: The Buffer Cache 

The buffer cache has two primary functions. The first, and most important, is to make possible concurrent I/O and 
computation within a UNIX process. The second is to insulate the user from physical block boundaries. 

From a user thread's point of view, I/O is synchronous. By this we mean that when the I/O system call returns, the 
system no longer needs the user-supplied buffer. For example, after a write system call, the data in the user buffer 
has either been transmitted to the device or copied to a kernel buffer — the user can now scribble over the buffer 
without affecting the data transfer. Because of this synchronization, from a user thread's point of view, no more 
than one I/O operation can be in progress at a time. Thus user-implemented multibuffered I/O is not possible (in a 
single-threaded process). In OSF/1, however, the user can utilize multiple threads within a task to program 
concurrent I/O and computation. 

The buffer cache provides a kernel implementation of multibuffering I/O, and thus concurrent I/O and 
computation are possible even for single-threaded processes. 
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5-17. The Buffer Cache 
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Student Notes: Multi-BufTered I/O 

The use of read-aheads and write-behinds makes concurrent I/O and computation possible: if the block cunently 
being fetched is block i and the previous block fetched was block i-1, then block z+i is also fetched. Modified 
blocks arc nomially not written out synchronously but are instead written out sometime after they were modified, 
asynchronously. 
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5-18. The Buffer Cache 
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Student Notes: Maintaining the Cache 

Active buffers are maintained in least-recentiy-used (LRU) order in the system-wide LRU list. Thus after a buffer 
has been used (as part of a read or write system call), it is returned to the end of the LRU list The system also 
maintains a separate list of "ft'ee" buffers called the aged list. Included in this list are buffers holding 
no-longer-needed blocks, such as blocks ftom tmncated files. 

Fresh buffers are taken from the aged list If this list is empty, then a buffer is obtained from the LRU list, as 
follows. If the first buffer Qeast recently used) in this list is clean (i.e., contains a block that is identical to its copy 
on disk), then this buffer is taken. Otherwise (i.e., if the buffer is dirty), it is written out to disk asynchronously 
and, when written, is placed at the end of the aged list. The search for a fresh buffer continues on to the next 
buffer in the LRU list, etc. 

When a file is deleted, any buffers containing its blocks are placed at the head of the aged list Also, when I/O 
into a buffer results in an I/O error, the buffer is placed at the head of the aged list 

In BSD, buffers that have been read (or written) in their entirety are placed at the end of the aged list The 
assumption is that, since files are normally accessed sequentially, these buffers won't be needed for a while. This 
technique has not been foimd to improve performance and thus is not used in OSF/1. 
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5-19. The Buffer Cache 
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Student Notes: Accessing the Cache 

Buffers in the cache are accessed via a hash table. In older versions of UNIX, buffers in the cache were identified 
by file-system number and block number (within the file system), ^th remote file systems such as NFS, the 
client does not know the block number within the file system, but only knows the block number relative to the 
beginning of the file. OSF/1 thus uses the address of the vnode and the block number relative to the beginning of 
the file to identify blocks of files of not only remote but also local file systems. 

This approach does not work for the indirect blocks and other metadata structures of UNIX file systems (both S5 
and UPS). These are identified by the address of the vnode of the underlying file system (i.e. block special file) 
and the block number relative to the beginning of the file system. (A possible consistency problem that would 
arise when blocks of open files in a mounted file system arc accessed via the block special interface is prevented 
by not allowing the block special interface to a mounted file system to be accessed.) 

In order to improve the performance of operations such asjsync.that affea the cached blocks of a particular file, 
each vnode heads a list of incore clean buffers and incore dirty buffers. 
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5-20. The Buffer Cache 



Virtual Buffers 
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Student Notes: Virtual Buffers 

If buffers were all of the same size and files were allocated in fixed-size blocks, then allocating a buffer would be 
trivial. However, the UFS file system allows different file systems to have different block sizes and, within a file 
system, it allows the last block of a file to be smaller than the others. 

Each ftwf stmcture is assigned a maximum-block-size amount of virtual memory (MAXBSIZE = 8K) for its 
buffer. 

The total amount of real memory allocated for buffers is divided up among the few/ structures; a possible result is 
that not all buffers wiU have the maximum 8K of real memory backing them up. If such an underendowed buffer 
is allocated when a fiill allotment of real memory is needed, space is "stolen" from another Z?m/ structure's buffer 
(by remapping the memory). Bi/ structures without real memory for their buffers are placed on an empty list. If a 
few/ structure is allocated whose buffer is larger than is needed, its extra space is given to a ^m/" structure on the 
empty list. 
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5-21. The Buffer Cache 



File System Consistency, part 1 
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Student Notes: File System Consistency, part 1 

In the event of a crash, the contents of the file system may well be inconsistent with any view of it the user might 
have. For example, a programmer may have carefully added a node to the end of the list, so that at all times the 
list stmcture is well-formed. 
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5-22. The Buffer Cache 



File System Consistency, part 2 
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Student Notes: File System Consistency, part 2 

But, if the new node and the old node are stored on separate disk blocks, the modifications to the block containing 
the old node might be written out first; the system might well crash before the second block is written out. 
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Keeping It Consistent 
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Student Notes: Keeping It Consistent 

To deal with this problem, system data structures are written out synchronously and in the correct order (i.e., the 
block containing the target of a pointer is updated before that containing the pointer). This is done for directory 
entries, inodes, indirect blocks, etc. 

No such synchronization is done for user data structures: not enough is known about the semantics of user 
operations to make this possible. However, a user process called update executes a sync system caU every 30 
seconds, which initiates the writing out to disk of all dirty buffers. Alternatively, the user can open a file with fl ie 
synchronous option so that all writes are waited for, i.e. th e buffer cache acts as a write-through c asiiQ (N.B. that 
this IS expensive!). 



5-49 



Module 5 — File Systems 



5-24. The Buffer Cache 



Parallelizing the Buffer Cache 

Locks: buf structure (blocking) 

free lists (LRU and aged) (spin) 
hash chains (spin) 

Precedence: buf stnicture > free list 

buf stnicture > hash chain 
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Student Notes: Parallelizing the Buffer Cache 

The buffer cache is parallelized by using blocking locks on the buffers. Thus many operations may proceed 
simultaneously, as long as they involve different buffers. To avoid race conditions when updating the free lists 
and hash table, spin locks are employed. A partial precedence order on these locks is used, as shown on the slide. 
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5-25. The Buffer Cache 



Block I/O Read 
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Student Notes: Block I/O Read 

Note that the use of events avoids the race condition between the biodone and the biowaiv. the intemipt could be 
handled on a different processor from the one on which the thread calling biowait is running. 
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5-26. The Buffer Cache 



Block I/O Read (Pseudocode) 

brcad(vnode, blkno) 

buffer = getblk(vnode, blkno) 

if (event_posted(bufFer->b_iocomplete)) 

returii(buffer) 
VOP_STRATEGY(vnode. buffer) 
event_wait(buffer->b_iocomplete) 
return(buffer) 
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Student Notes: Block I/O Read (Pseudocode) 
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5-27. The Buffer Cache 
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Student Notes: Finding a Block in the Cache 

First the thread takes the lock on the hash chain header (simple lock). If it finds the desired buffer, then it unlocks 
the header and takes the lock on the buffer (blocking lock). If, after the thread waits for the lock, it finds that the 
buffer no longer contains the desired block, then the thread repeats the procedure from the beginning. 



i 
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5-28. The Buffer Cache 



Finding a Block in the Cache (Pseudocode) 



restart: 



getbIk(vnode, blkno) 

hash(vnode, blkno) 
Iock(hash chain) - 
for each buffer in chain { 

if (buffer.blkno == blkno) { 
unlock(hash chain) 



lock(buffet) 

if (buffer.blkno ^ blkno) { rec^^ 

\mlock(buffer) 

goto restart 

} 
return(bufiFer) 



} 



} 



/* block is not in cache */ 
unlock (hash chain) 



buffer = getnewbuf( ) 

/* has someone else just now allocated a different 
buffer for the same block? */ 

^ lock(hash chain) 

/if (hash chain has been modified) { /^(check 
timestamps) */ 

if (blkno is in hash chain) { 
return buffer to free list 
xmlock (hash chain) 
goto restart 
} 
} 

insert (buf , hash chain) 
unlock (hash chain) 
retuni(buffer) 
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Student Notes: Finding a Block in the Cache (Pseudocode) 

Associated with each hash chain is a timestamp that is incremented by one when the hash chain is modified (a 
buffer is either inserted or removed). 

There is a potential race condition when getnewbuf is called: two threads may simultaneously discover that a 
particular block is not in the cache, and both call getnewbirfxo allocate a buffer for it (and two buffers are indeed 
allocated). Due to the lock on the hash chain, one buffer will be inserted in the hash table first To prevent both 
buffers (representing the same block) from being inserted, a check has to be made to insure that the buffer being 
inserted is not a duplicate. This check would involve searching the hash chain (again). To minimize the number 
of times this must be done, the current value of the timestamp on the hash chain is compared with its value when 
it was originally ascertained that the block was not present. Only if the timestamps are now different is the hash 
chain searched. 
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5-29. The Buffer Cache 



Getting a New Buffer 
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Student Notes: Getting a New Buffer 

First the thread takes the lock on the header (simple lock). Then it conditionally takes the lock on the buffer. If 
the buffer is already locked, then the thread skips it and tiles the next one. If no buffers are available, then the 
thread sleeps until one is. 
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5-30. The Buffer Cache 



Getting a New Buffer (Pseudocode) 

getnewbuf( ) 

lock (free list) 
for each buffer in free list { 
if Gock_try (buffer)) 
break; 

} 

remove buffer from free list 

unlock (free list) 

event_clear (buffer->b_iocomplete) 

return(buffer) 
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Student Notes: Getting a New Buffer (Pseudocode) 
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5-31. The Buffer Cache 



Mmap 



address space 
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student Notes: Mmap 

The mmap system call is used either to map a Hie into a process's address space or to create an anonymous 
memory region. Anonymous memory is shared with all of the process's descendants. 

A mapped file may b e private, meaning that changes to the mapped memory are not shared with other processes 
and are not refle cted back to the file. 

A mapped file may be shared, meaning that changes to the mapped memory are shared with other processes that 
have a shared mapping of the file, and these changes are reflected back to the file . 

Two important issues arise with mmap. First, does a process that has a private mapping of a file "see" the 
changes made by processes with shared mappings? In O SF/1 , the answer isng. 

The other issu e involves the simultaneous access of a file vi a mmap andjgad/ Write system calls. In the current 
implementation there is a consistency problem, since two copies of blocks of the file may exist in primary 
memory: one in the buffer cache and one in a page frame to which a virtual page has been mapped. 
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5-32. Directory Path Searching 



Directory Path Searching 



start with root vnode or current-directory vnode 
while (not at end of path) { 

search for next component in file represented by current vnode 

if not found 
temiinate 

fetch associated vnode, assign it to current vnode 
} 
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Student Notes: Directory Path Searching 

Following directory paths would seem to be quite trivial. The basic algorithm is shown in the picture. However, 
as will be discussed, the actual procedure is fairiy complex, and this subsystem is a very important part of the 
operating system. 
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5-33. Directory Path Searching 



Complications in Directory Path Searching 
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Concurrency 
Speed 
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Student Notes: Complications in Directory Path Searching 
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5-34. Directory Path Searching 



Multiple File Systems 



• The top-level path-searching routine is namei 

• Namei breaks the path into components and, for each component, calls the 
appropriate file system (via VOPJLOOKUP) to look it up in the current 
directory 
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Student Notes: Multiple File Systems 

One might think that a more efficient technique for following a path would be to give the file system lookup 
routine all of the remaining portion of the path so that it can follow it as far as possible. This technique is not 
easy, however, for a number of reasons. 

In NFS, it is up to the client to detennine which character separates components; the server is not involved. For 
example, UNIX clients use *'/" as the component separator, whereas MS-DOS uses A". Only the client can break 
a pathname into its components (though one might argue that the client could pass the component-separator as an 
argument to the server). But, furthermore, mount points are interpreted strictly by the client, and server mount 
points mean nothing to the client 
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5-35. Directory Path Searching 



Mount Points 
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Student Notes: Mount Points 

Mount points are encoded in the vnode and mount stnictures. This system makes it not only possible but 
obligatory that clients view a file system independent of mounts done by the server. Namei tests each directory it 
encounters to determine if it is a mount point; if it is, namei calls the mounted file system's VFS_ROOT routine to 
obtain its root vnode. 

A related scenario is following ".." links out of a mounted file system. In this case, namei consults the mounted 
file system's mount structure to find the address of the vnode that it covers, and then it follows that directory's ".." 
link of the directory represented by that vnode. 
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5-36. Directory Path Searching 



Symbolic Links 
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Student Notes: Symbolic Links 

If a vnode maiked as a symbolic link is encountered, then the file system's VOP_READLINK routine is called to 
get the link. The routine replaces that portion of the path that has already been followed with the value of the 
symbolic link, and then restarts the search firom the beginning of the newly modified path name. 

To avoid loops caused by careless placement of symbolic links, no one path may be composed of more than 
MAXSYMLINKS (32) symbolic links. 
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5-37. Directory Path Searching 



Concurrency 



Must guard against two types ofracg conditions: 

• a directory is modified while it is being searched 

• a directory is modified after a lookup, but before the result is acted upon 
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student Notes: Concurrency 

OSF/l deals with the first case by requiring that a thread hold a read lock on a directory while searching it and 
hold a write lock on the directory while modifying it. 

Ti mestarnp s are used to deal with the second case. Each time a thread modifies a directory, it increments the 
directory*s timestamp by one. When a thread searches a directory it records the directory's current timestamp. 
Before the thread modifies a directory, it compares the timestamp it obtained in the lookup request to the 
directory's current timestamp. K the timestamps are different, then the directory must have been modified since 
the lookup, and the thread repeats the lookup. If the lookup fails a second time, then the operation fails. 

For example, suppose two threads issue concurrent delete requests for the same directory entry. The net result 
should be that one succeeds and the other fails. Both threads do a successfiil lookup of the entry and one thread 
succeeds in deleting the entry. The other thread will note before it attempts to remove the entry that the timestamp 
has changed. It will thus repeat the lookup, the lookup wiU fail, and so the delete system call will fail. 

An example of the destractive effects of the second race condition is the following set of operations, each 
performed by a separate thread: 

thread 1: rm /A 

thread 2: rtn /A 

thread 3: cp/C /B 

The two rms are executed concurrently: both threads do a successful lookup to determine that /A exists; as a side 
effect the lookup returns the position of the component A within the / directory. This lookup is performed while 
holding a read lock; thus botti threads can do it in parallel. Modifying the directory to delete the entry A, 
however, requires an exclusive write lock. Thus one thread blocks while the other thread removes the entry A. 
However, it happens that immediately after thread 1 removes entry A, thread 3 creates the entry B in the directory 
slot just vacated by A. When thread 2 wakes up and completes its operation, it removes what it thinks is entry A 
but is in fact entry B. 
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5-38. Directory Path Searching 



Speed 



Fancier file-naming facilities result in longer lookup times 
Solution: more caching 
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Student Notes: Speed 

4.2BSD added many new facilities not present in earlier versions of UNIX. One result of these additions was that 
4.2BSD was considerably slower than 4.1BSD. Kernel profiling showed that approximately 25% of system time 
spent in the kernel was spent in routines ganslating directory paths. This was much too much time for such 
chores, so to speed things up, two forms of caching were introduced. Both forms were designed for use with the 
UFS file system, but may be used with any file system. With the addition of the two types of caching, the system 
time devoted to name translation dropped from 25% to less than 10%. 
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5-39. Directory Path Searching 



The Lookup Cache 
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Student Notes: Lookup Cache 

The l ookup cache is a cache of the mos t recent component-name'to-vnod e translatioi^ . Searching a directory for 
a component name can be expensive, so the most recent lookups are kept in a cache. (Note that this is not a cache 
of path names, but merely of component names.) 

A vnode reference presents a problem in representing the result of a translation. If the cache contains actual 
"reference-counted" references to the vnode, then the reference count on vnodes themselves remains positive, and 
incore vnodes are not freed. (SVR4 actually does employ this technique: when the system is low on available 
vnodes, it makes a pass through the cache and frees those vnodes whose only reference is due to the cache.) 

The OSF/1 cache, derived from 4.4BSD, contains "soft" references to vnodes, i.e. references that do not show in 
the reference count. The problem here is that if a file is deleted and its vnode reused for another file, the cache 
continues to contain a reference to the vnode in its previous incarnation, since there is no indication in the vnode 
that a cache entry refers to it. To deal with this, vnodes and the cache contain c(mdbUU ks (version 
juiffibere) — 32-bit integers. Each vnode has an assigned capability. When the vnode is invalidated, the version 
number is incremented by one. Each cache translation also contains a version number, which is set equal to that 
of the vnode. If the version numbers do not match when the translation is accessed, then the cac he entry is 
considered invalid and is flushed. Berkeley's figures indicate that this cache has a "hit rate" of(t0-80^^ 
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5-40. Directory Path Searching 



The Search Cache 
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Student Notes: The Search Cache 

The second fonn of caching deals with repeated lookups of one directory. Consider a command such as Is -I: its 
implementation involves reading the contents of the directory, then perfonning a stat ^system cal l on each entry. It 
takes time proportional to i to search for the I'th entry, since the search always starts at the beginning of the 
directory. Thus, for n entries , time proportional to rP - is needed to find each entry in the directory. For a large 
directory this could be rather significant. 

By storing in the inode the offset of where the last search terminated, a linear algorithm for Is -I (and others) can 
be devised, since the search for the next item in the directory will start where the previous item was found. 

Note that 4.2BSD and 4.3BSD stored the offset in the user structure. This approach seems better, especially if 
multiple threads are each doing the equivalent of Is -I on the same directory. However, storing the offset in the 
inode makes this technique work when Is -I is applied to directories other than the current directory (i.e., when 
each directory lookup involves searching a path). 

The search cache has a "hit rate" of 5-15%. 
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5-41. 85 File System 



The S5 File System 



The original UNIX file system: 

• extremely simple 

• no attempt to optimize the layout of files 
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student Notes: The S5 File System 

The S5 file system, provided primarily for gnrnpatihility rp-^RnnSj is generally always slower than the UFS file 
system. However, it has a few things in common with the UFS file system, in particular the notion of inodes 
(including the disk map). 
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5-42. S5 File System 
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Student Notes: Inodes 

Inodes arc the focus of all file activity, i.e., every access to a file must go through the inode. Every file has a 
inode on permanent storage; this on-disk inode is of type struct dinode in the S5 file system. All open files, 
current directories, mounted-on directories, and the root have incore inodes of type struct SSinode. Once brought 
into primary storage, an inode stays there until its associated file is deleted or its storage is needed for some other 
purpose. 
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5-43. S5 File System 
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Student Notes: Disk Map 

The purpose of the disk-map portion of the inode is to map block numbers relative to the beginning of a file into 
block numbers relative to the beginning of the file system. An S5 file system may be configured with aM^^byte, 
IK-byte , o r 2K-by te block size. We assume a IK block size from here on. 

The disk map consists of 13 pointers to disk blocks, the first 10 of which point to the first 10 blocks of the file. 
Thus the first 10Kb of a file are accessed directly. If the file is larger than 10Kb, then pointer number 10 points to 
a disk block called the indirect block. This block contains up to 256 (4-byte) pointers to data blocks (i.e., 256Kb 
of data). If the file is bigger than this (256K +10K = 266K), then pointer number 1 1 points to a double indirect 
block containing 256 pointers to indirect blocks, each of which contains 256 pointers to data blocks (64Mb of 
data). If the file is bigger than this (64Mb + 256Kb + 10Kb), then pointer number 12 points to a triple indirect 
block containing up to 256 pointers to double indirect blocks, each of which contains up to 256 pointers pointing 
to single indirect blocks, each of which contains up to 256 pointers pointing to data blocks (potentially 16Gb, 
although, as will be discussed, the real limit is eithe r 2Gb or 4Gb). 

The structure of t he UF S^file system is similar, except that the block size is eithe r 4K or 8K a nd the disk map 
consists of 15 pointers, the first 12 of which point to the first 12 data blocks. Because of the larger block size, the 
triple indirect block is unusable, since the double indirect block can represent a file size larger than 4Gb. A hard 
limit on file size for 32-bit architectures is 4Gb (or periiaps 2Gb, depending on one's feelings about sign bits), 
since the offset into a file must fit in a word! 

This data strucmre allows the efficient representation of sparse files, i.e., files whose content is mainly zeros. 
Consider, for example, the effect of creating an empty file and then writing one byte at location 2,000,000,000. 
Only four disk blocks are allocated to represent this file: a triple indirect block, a double indirect block, a single 
indirect block, and a data block. All pointers in the disk map, except for the last one, will be zero. If the file is 
read, all bytes up to the last one will read as zero. This is because a zero pointer is treated as if it points to a block 
containing all zeros: a zero pointer to an indirect block is treated as if it points to an indirect block fiUed with zero 
pointers, each of which is treated as if it points to a data block filled with zeros. However, one must be carefiil 
about copying such a file, since commands such as cp and tar actually attempt to write all the zero blocks! (The 
dump command, on the other hand, copes with sparse files properly.) 

The units of the pointers in the disk map in the S5 file system are in blocks (IK). For the UFS file system, the 
units are in fragments that can be any multiple of 512 bytes, fix)m 512 bytes to 8K bytes (this value is fixed for 
each instance of the file system). 
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The S5 dircaory consists of an array of pairs of inode number and component number. An important restriction is 
that the component name may be no longer than 14 bytes, thereby making a fixed length fonnat possible. Note 
that identifying a file requires a reference to the file system as well as the inode number, but only the latter is 
supplied in each directory. The file system is assumed to be the one that contains the directory. Thus the only 
way a path can cross a file system boundary is via mount points. 



5-93 



Module 5 — File Systems 



5-46. S5 File System 



File System Layout 
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• Bootblock 

— used on some systems to contain a bootstrap program 

• Superblock 

— describes the file system: 

♦ total size 

♦ size of inode list (I-list) 

♦ header of free-block list 

♦ list of finee inodes 

♦ modified flag 

♦ read-only flag 

♦ number of free blocks and free inodes 

♦ resides in a buffer borrowed from the buffer cache while the file system is mounted 

• I-Ust 

— area for allocating inodes 

• Data region 

— remainder of file system is for data blocks and indirect blocks 

A problem with this organization is that the I-list and the data region are separated from each other. Since one 
must always fetch the inode before reading or writing the blocks of a file, the disk head is constantly moving back 
and forth between the I-list and the data region. 
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Free disk blocks are oiganized as shown in the picture. The superblock contains the address of up to NICFREE (= 
100) free disk blocks. The last of these disk blocks contains NICFREE pointers to additional free disk blocks. 
The last of these pointers points to another block containing up to NICFREE free disk blocks, etc., until all free 
disk blocks are represented. Thus most requests for a free block can be satisfied by merely getting an address 
from the superblock. When the last block reference by the supeiblock is consumed, however, a disk read must be 
called to fetch the addresses of up to 100 more free disk blocks. Freeing a disk block results in reconstructing the 
list structure. 

This organization, though very simple, sc atters the blocks of files all over the s urface of the disk . When allocating 
a block for a file, one must always use the next block fix)m the ftSe list; there is no way to request a block at a 
specific location. No matter how carefully the free list is ordered when the file system is initialized, it becomes 
fairly well randomized after the file system has been used for a while. 
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Inodes are allocated from the I-list. Free inodes are represented simply by zeroing their mode bits. The 
superblock contains a cache of indices of free inodes in an array called sjnode (of size NICINOD). When a finee 
inode is needed (i.e., to represent a new file), its index is taken from this cache. If the cache is empty, then the 
I-list is scanned sequentially until enough free inodes are found to refill the cache. 

To speed this search somewhat, the cache contains a reference to the inode with the smallest index that is known 
to be fiiee. When an inode is free, it is added to the cache if there is room, and its mode bits are zeroed on disk. 



5-99 



Module 5 — File Systems 



5-49. UFS File System 



The UFS File System 



The goal is t o lay out files on disk so that they can be accessed a s quickly 
as possible and so that no more than a minimal amount of disk space is 
wasted 

ComponenLnames jjf directories can be much longer than in the S5 file 
system 

Fully parallelized ^ 



^■*9- ©1990. 1991 Open SoftwueRrarnktioii 



5-100 



Module 5 — File Systems 



Student Notes: The UFS File System 



5-101 



Module 5 — File Systems 



5-50. UFS File System 



UFS Directory Format 
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UFS allows component names to be up to 255 characters long, thereby necessitating a variable-length field for 
components. Directories arc composed of 512-byte blocks and entries must not cross block boundaries. This 
design adds a degree of atomicity to dircaory updates. It should take exactl y one disk . writg .t o update a director y 
entry (512 bytes was chosen as the smallest conceivable disk sector size). If it takes two disk writes to modify a 
directory entry, then clearly th e disk will crash between the two disk writes! 

Like the S5 directory entry, the UFS directory entry contains the inode number and the component name. Since 
the component name is of variable length, there is also a string length field (the component name includes a nuU 
byte at the end; the string length does not include the null byte). In addition to the string length, there is also a 
record length, which is the length of the entire entry (and must be a multiple of four to ensure that each entry starts 
on a four-byte boundary). The purpose of the record length field is to represent free space within a directory 
block. Any fiee space is considered a part of the entry that precedes it, and thus a record length longer than 
necessary indicates that fi^ee space follows. If a directory entry is fiiee, then its record length is added to that of the 
preceding entry. However, if the first entry in a directory block is fi-ee, then this free space is represented by 
setting the inode number to zero and leaving the record length as is. 

Compressing directories is considered to be too difficult. Free space within a directory is made available for 
representing new entries, but is not returned to the file system. However, if there is free space at the end of the 
directory, the directory may be truncated to a directory block boundary. 
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5-51. UFS File System 



How to Do Disk I/O Quickly 



1. Transfer as muc h as possible with each I/O request 

2. Minimize seek time (i.e. reduce head movement) 

3. Minimize latency time 
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The UFS file system uses three techniques to improve I/O performance. The first technique, which has perhaps 
the greatest payoff, maximizes the amount of data transferred with each I/O request by using a relatively large 
block size. UFS block sizes may be eith er 4K bytes o r 8K bytes (the size is fixed for each individual file system). 
A problem with using a large block size is the wastage due to internal fragmentation: on the average, half of a 
disk block is wasted for each file. To alleviate this problem, blocks under certain circumstances may be shared 
among files. 

The second technique to improve performance is to minimize seek time by attempting to locate the blocks of a file 
so that they are near to one another. 

Finally, UFS attempts to minimize latency time, i.e. to reduce the amount of time spent waiting for the disk to 
rotate to bring the desired block underneath the desired disk head (many modem disk controllers make it either 
impossible or unnecessary to apply this technique). 
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• Superblock fj/rucr/^^ 

— incore while the file system is mounted 

— contains the parameters describing the layout of the file system 

— for paranoia's sake, one copy is kept in each cylinder group, at a rotating track position 

• Cylinder group summary (struct csum, one for each cylinder group) 

— incore while the file system is mounted 

— contains a summary of the available storage in each cylinder group 

— allocated fi^om the data section of cylinder group 

• Cylinder group block (struct eg) 

— resides in the buffer cache "as needed" 

— contains free block map and all other allocation infonnation 

Note: the superblock contains two sorts of infonnation, static and dynamic. The static information describes the 
layout of the entire file system and is essential to make sense of the file system. The dynamic infonnation 
describes the file system's current state and can be computed from redimdant infonnation in the file system. If the 
static portion of the superblock is lost, then the file system cannot be used. To guard against this, each cylinder 
group contains a copy of the superblock (just the static infonnation needs to be copied). 

A possible (though imlikely) failure condition might be that the entire contents of one surface are lost, but the 
remainder of the disk is usable. However, if this surface contains all copies of the superblock, then the rest of the 
disk would be effectively unusable. To guard against this, the copy of the superblock is placed on a different 
surface in each cylinder group. Of course, the system must keep track of where these copies are. This 
information is kept in the disk label (along with information describing how the physical disk is partitioned). 



5-107 



Module 5 — File Systems 



5-53. UFS File System 



Minimizing Fragmentation Costs 



5-53. 



• A file system block may be split into fragments that can be independently 
assigned to files 

— fragments assigned to a file must be contiguous and in order 

• The number of Augments per block (1, 2, 4, or 8) is fixed for each file 
system 

• Allocation in fragments may only be done on what would be the last M ock-- 
of a file, and only if the file does not contain indirect blocks 
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The Use of Fragments, part 1 
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This example illustrates a difficulty associated with the use of fragments. The file system must preserve the 
invariant that fragments assigned to a file must be contiguous and in order, and that allocation of fiagments may 
be done only on what would be the last block of the file. In the picture, the direction of growth is downwards. 
Thus file A may easily grow by up to two fragments, but file B cannot easily grow within this block. 

In the picture, file A is 18 fragments in length, file B is 12 fragments in length. 
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The Use of Fragments, part 2 





' .jjfjj^.^.jr 






AVVVkkT 



SS2SS 



^^ 



file A 



fileB 



5-55. 



* 1990, 1991 Open Softwm FmnKktion 



5-112 



Module 5 — File Systems 



Student Notes: The Use of Fragments, part 2 
File A grows by one fragment. 
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The Use of Fragments, part 3 
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File A grows by two more fragments, but since there is no space for it, the file system allocates another block and 
copies file A's fragments into it How much space should be available in the newly allocated block? If the newly 
allocated block is entirely ftee, i.e., none of its fragments are used by other files, then further growth by file A will 
be very cheap. However, if the file system uses tMs approach all the time, then we do not get the space-saving 
benefits of fragmentation. An alternative approach is to use a "best-fit" policy: find a block that contains exactly 
the number of free fragments needed by file A, or if such a block is not available, find a block containing the 
smallest number of contiguous fi"ee fragments that will satisfy file A's needs. 

Which approach is taken depends upon the degree to which the file system is fragmented. If disk space is 
relatively unfragmented, then the first approach is taken ("optimize for time"). Otherwise, i.e., when disk space is 
fragmented, the file system takes the second approach ("optimize for space"). 

The points at which the system switches between the two policies is parameterized in the superblock: a certain 
percentage of the disk space, by default 10%, is reserved for superuser. (Disk allocation techniques need a 
reasonable chance of finding free disk space in each cylinder group in order to optimize the layout of files.) If the 
total amount of fragmented free disk space (i.e., the total amount of free disk space not counting that portion 
consisting of whole blocks), increases to 8% of the size of the file system (or, more generally, increases to 2% less 
than the reserve), then further allocation is done using the best-fit approach. Once this approach is being used, if 
the total amoimt of fragmented fiee disk space drops below 5% (or half of the reserve), then further allocation is 
done using the whole-block technique. 



5-115 



Module 5 — File Systems 



5-57. UFS File System 



Minimizing Seek Time 



• 




The principle: 

— keeprdated infonnation asdoseJogeflieraspossiWe 

— distribute information su fficiently to make the above possible 

The practice: 

attempt to put new inodes in the same cylinder group as thei r directory 
put inodes for new directories in cylinder groups with "lots" of free space 

— put the beginning of a file (direct blocks) in the inode*s cylinder group 

— put additional portions of the file ( each 2Mb) in cylinder groups with 
"lots" of free space 
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A naive way of laying out consecutive blocks of the file on a track would be to put them in consecutive locations. 
The problem with this is that some amount of time passes between tiie completion of one disk request and the start 
of the next. During this time, the disk rotates a certain distance, probably far enough so that the disk head is 
positioned after the next block. Thus it will be necessary to wait for the disk to rotate almost a complete 
revolution for it to bring the beginning of tiie next block underneath the disk head. This delay could cause a 
significant slowdown. 
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Minimizing Latency, part 2 
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A better technique is not to lay out the blocks on the track consecutively, but to leave enough space between them 
so that the disk will rotate no further than to the position of the next block during the time between disk requests. 

It may be that when a new block is allocated for a file, the optimal position for the next block is already occupied. 
If so, one may be able to find a block that is just as good. If the di^ has multiple surfaces (and multiple heads), 
then we can make the reasonable assumption that the blocks underneath each head can be accessed equally 
quickly. Thus the stack of blocks underneath the disk heads at one instant are said to be rotationally equivalent. 
If all of these blocks are occupied, then the next stack of rotationally equivalent blocks in the opposite direction of 
disk rotation is almost as good as the first. If all of these blocks are taken, then the third stack is almost as good, 
and so forth all the way around the cylinder. If all of these are taken, then any block within the cylinder group is 
chosen. 

This technique is perhaps not as useful today as in the past, since many disk controllers buffer entire tracks and 
hide the relevant disk geometry. 



5-121 



Module 5 — File Systems 



5-60. UFS File System 



ParallelizationofUFS 
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Two sorts of locking are used with UFS, blocking RW locks and simple locks (spin locks): 

1 . blocking RW locks (on inodes): used to protect the file across logical operations. I.e., synchronization is 
supplied at the granularity of the operations described by tdo structures. As a special case, eg blocks reside in 
the buffer cache and are locked via the blocking lock on the buffer from the cache. 

2. siny}le locks (spin locks): used to protect important system data structures (inodes, vnodes, and superblocks). 
Modifications to these data structures are always synchronized with simple locks. However, on many 
architectures, such synchronization is not necessary for reads: if the architecture guarantees that 32-bit, 
aligned items can be read atomically, then no locking is required. Thus, for example, a thread can read the 
mode bits from the inode and be guaranteed that they make sense. 

Parallel architectures that do not supply such atomicity guarantees arc deemed to have bogus memory. These 
cases are dealt with in the source code with the BM macro: BM(lock(x)) expands to lock(x) on bogus-memory 
machines and expands to the null string on other machines. Thus locking is compiled conditionally. 
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NFS Highlights 



Servers are stateless : 

• server crash recovery is trivial 

• NFS does no t support full UNIX semantics 

• NFS is "easily" supported on other operating systems 
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Since servers contain no infomiation about their clients, crash recovery is trivial in NFS: there is no infomiation to 
be recovered after a crash. However, some state infomiation is required for implementing certain UNIX I/O caUs, 
and thus NFS cannot duplicate UNIX semantics exactly. 

For example, a common technique for creating a temporary file is for a process to create a file and then to unlink 
the newly created file. Since the file is open, it continues to exist even though it has a zero link count (the 
reference count on its vnode is positive). The fil e is removed only when it is closed . " 

If this technique is practiced over NFS, the server does not know that the file is open (since this would be state 
information), and thus removes the file as it is unlinked. Since a number of important ^plications use this 
technique for creating a temporary file, the method must be accommodated. The client-side NFS code (executing 
in the kemel) conver ts unlink re quests mto_ rename requests, changing the name of the file to a temporary name. 
When the client application finally closes the file, the clos e is-COD ^erted into an unlink and the file is removed. 

Another example of the difference between UNIX and NFS semantics arises when an application changes the 
access permissions of an open file. Accesschecks for UNIX files are performed only when the file is opened. 
Thus, if the user successfiiUy opens a file for read-write access and subsequently changes the permissions to 
read-only, write access to the already open file is stiU allowed. However, since the NFS server must check access 
permissions with each access to a file, write access would be denied in this case. 

OSF/1 (and other UNIX implementations of NFS) provides only a partial solution to this problem. The NFS 
server aUows the owner of a file read-write-execute access regardless of the permissions associated with the file; 
the NFS client filters requests to the NFS server on the basis of how the file was opened. Thus if the file was 
opened successfully for read-write access, then the client side allows read and write calls to be processed. 
However, if the file was opened as read-only, then the client side denies write requests. 

A further difference between UNIX and NFS semantics is caused by the fact that NFS clients cache blocks fi-om 
files provided by NFS servers. This means that processes on different machines do not necessarily have a 
consistent view of shared files. 
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The client and server communicate via Sun*s RPC protocol. The XDR protocol copes with the heterogeneous 
environment. The two major issues arc reliability and security. 

The transport protocol is typically UDP , an unreli able protocol. Thus NFS itself must provide reliability. NFS 
accomplishes this by taking advantage of the request/response semantics of the client-server interaction. 

For example, suppose that a client issues a -write request but receives no response. The client will repeat the 
request under the assumption that the original request was lost. 

However, suppose that it was the response that was lost, and not the request. Now the server receives the -write 
request twice. This usually presents no problems, because most NFS requests, such as wnYe, sr cAdempotent . 
meaning that t he effect of perfomiing the request twice i s the same as performing it onc e. The write request is 
idempotent since it contains the location in the file to which the data is to be written. However, there are 
additional problems with reliability, as will be seen. 

Security has always been a problem in NFS. The model for authentication is essentially 'Trastme." Each NFS 
RPC request contains as part of its header the numeric user id of the caller. Servers refuse requests from the 
superaser but will trastingly honor any other requests. Sun uses an enhanced authentication t echnique for RPC 
involving a combination o f PES and public-key encryptio n. OSF will deal with such problems through its 
distributed computing environment (DCE). 
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File Handles 
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When a file is opened, it is identified by its path name. The NFS server verifies that the file exists, checks that the 
desiied access is currently allowed, and returns SifiUhandk that the client will use to identify the file on 
subsequent accesses. Using the file handle for subsequent accesses thus avoids expensive path traversal for each 
access. This file handle is of an opaque data type meant not to be interpreted by the client but only to be passed 
back to the server. 

UNIX servers pass back a handle consisting of: 

• file system number ---^ 

• inode number ^-" 

• inode generation number -^ 

The generation number copes with the confusion that could arise finom the reuse of inodes. One client may open a 
file, another delete it, and a third might reuse the inode when it creates an entirely new file. When the first client 
attempts to access the original file, the server must be able to determine that the desired file no longer exists. So, 
when a client reuses an inode, the inode receive s a new generation number to ctistinffliish its current use from past 
uses. When a client accesses a no-longer-extant file, a "stale file handle" error message is returned to the client. 



The generation number is stored on disk in the inode. 
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Client-Side Caching 
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Remote disk blocks arc cached in the client's buffer cache. If multiple clients use the same file, therc may be a 
consistency problem. While it is considered too expensive to keep the various caches consistent, an attempt is 
made to keep things fiom being too inconsistent In each nfsnode is a copy of the associated remote file's 
attributes (i.e., what is obtained from a Stat s ystem call — ^infonnation such as the file's modification time). Every 
time th e attribute s are fetched from the server, an expiration time of some number of seconds is set (five seconds 
in OSF/1). If the file is accessed before the attributes expire, then it is assumed that any locally cached blocks of 
the file arc valid. If the attributes have expircd, the new attributes must be obtained from the server and, if the file 
has been modified, then the locally cached blocks arc flushed. (Modified cached blocks are written to the server.) 

The cache is cleaned in response to close, sync, and fsync system calls (fsync is performed sjnridironously^over 
NFS). 
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nfsbiod Processes 



• Concurrent I/O and computation require asynchronous read-aheads and 
write-beMnds 



• NFS *s RFC requests are synchronous 

• Solution: use additional kernel threads 
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When a process accesses files through the buffer cache, concurrency between 1/0 and computation is achieved by 
exploiting read-aheads and write-behinds. This is easy to do for 1/0 for loc al files^ because the interface to the 
device driver is asynchronous . For example, when reading a file sequentially, one can start I/O read requests for 
the current block and the next block without waiting for the request to the latter. When writing a block, the user 
process merely modifies the buffer cache and the file itself is modified later, asynchronously. 

The interface between the client and the NFS server is svnchrono us. since RFC requests are inherently 
synchronous. To achieve the desired concurrency, separate threads are used on the client to perform many NFS 
client RFC caUs. These threads are pre-created and are known a s ftfsbiod processes ^(these are user processes that 
have executed the async__daemon system call). Whenever an asynchronous I/O request is desired, the client 
checks to see jf an ftfsbiod process is^available. If so, fli en the request is given to it to perfo rp^ in its nwn mntp.xt 
Otherwise, the caller performs the request in the caller's context (and blocks until the request is completed). 
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5-67. NFS File System 



nfsd Processes 



• Kernel processes that handle NFS requests on the server 

• Must deal with "authentication" and access checking 
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Student Notes: nfsd Processes 

Each server has a number of r^sd processes that handle the incoming RPC requests for NFS. (These are user 
processes that have executed the nfssvc system call.) Unlike the «^Worf processes, the n/j^ processes are 
essential. NFS requests arc handled only in their context on the server. When such a process receives a request, it 
acts on behalf of the caller and temporarily assumes its identity. This is accomplished through the use of 
credentials structures, which contain groupids and a userid and are passed to the access-checking routines. 
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5-68. NFS File System 
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Student Notes: Server's Buffer Cache 

The server's buffer cache is used for handling client requests, but it is treated as a write-thro uph cache : when an 
nfsd process handles a write request, not only is the cache modified but also the data is wntten to the disk 
immediately and the RPC call does not return until the disk-write completes. This technique is consistent with the 
idea that NFS servers are stateless: data that is in the cache but not on disk is state information that the client 
would not want the server to lose if the server were to crash. The client is assured that, when an NFS RPC request 
remms, any requested changes to a file have been reflected on disk. 
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5-69. NFS File System 
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Student Notes: The NFS Mount Protocol 

Like local file systems, in UNIX a remote file system must be mounted in the client's directory hierarchy in order 
to be used. 

In OSF/1, the mount sheU command makes an RPC request to the server's mountd process to obtain a file handle 
for the mount point. The mountd process is a user process that implements the server side of the mount protocol 
(the mount shell command implements the client side). Each server maintains in the /etc/ exports file a list of 
exported file systems and the clients to which they arc exported. The mountd process first makes certain that the 
client is allowed to mount the requested file system, then rctums to it the file handle for the root of the file system. 
The mount sheU command then issues a mount system call, passing to the kernel the file handle and the path 
name of the moimt point. 
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5-70. NFS File System 



Remote M ounting, part 1 
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Student Notes: Remote Mounting, part 1 



In this picture we have two machines, nancy and sluggo. Sluggo exports two file systems to nancy, identified as 
lusrlsrc and lusrlman. 



5-143 



Module 5 — File Systems 



5-71. NFS File System 



Remote Mounting, part 2 
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student Notes: Remote Mounting, part 2 

Nancy mounts sluggo*s lusrlman on its own lusrlman directory. 
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5-72. NFS File System 



Remote Mounting, part 3 
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Student Notes: Remote Mounting, part 3 

Nancy now mounts sluggo's lusrlsrc on nancy's lusrlosrc. On sluggo, lusrlsrclsys is a mount point: another file 
system, X, has been mounted here and, from sluggo*s point of view, the root directory of this file system is 
superimposed on top of the directory lusrlsrclsys (thus the original contents of this directory are invisible to 
sluggo). However, nancy does not see this mount point. 

The directory (on nancy) lusrlosrclsys is not mounted upon. Unlike sluggo, nancy sees the actual contents of this 
directory. If it is desired that this mount point exist in nancy's view as it does in sluggo's, then nancy could 
explicitly mount file system X on top of the directory lusrlosrclsys. 

The reasons for not having a client use the server's mount points arc partly for security, but mainly for simplicity. 
Suppose on server B, file system Y (from server Q is mounted on a directory within file system X and file system 
Z (on the same server as X) is also mounted within file system X. If client A mounts X (and thus appears in B 's 
/etc/exports list), what would be required for it to be able to follow the mount point to Y on server C? A must 
appear in C's /etc/exports list. But C has only verified that B is there. If B were to pass on A's requests to C, it 
would have to ensure that C approves of A. Rather than do this complicated checking, the convention is that A 
must mount Y itself. 

Note that it wouldn't be very difficult for B to allow A to follow the mount point to Z, but, again, for simplicity, 
this is not done. 
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5-73. NFS File System 



When the Server Crashes ..• 



• Hard mounts 



• Soft mounts 
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student Notes: When the Server Crashes ... 

The client's response to server crashes depends upon an option specified when the remote file system was 
mounted. If the client specified a hard mount, then any system call involving a file on the remote machine blocks 
until the machine comes back up (whether this takes seconds or weeks). Such system calls block uninterruptibly, 
so there is no way to abort the process making the system call. This can be very annoying. 

Another option is the soft mount . Any system calls involving files on the dead remote machine will return 
(eventually) with the error cod e ETIMEDOUT, This option might seem a good idea, but there are difficulties. A 
number of UNIX applications pay no attention to enor returns on I/O system calls (if the open succeeded, there 
could not possibly be any problems with reads and writes...). Thus damage may be done because the client is 
unaware of the crash. 

A more reasonable way of mounting the remote file system is the interruptible hard mount . With this option, as 
before, system calls involving a file on the remote machine block until the machine comes back up, but the wait is 
interruptible (i.e., by signals). However, the interrapt is not immediate: the undeilying RPC layer performs many 
retries before checking to see if a signal is pending. 
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5-74. NFS File System 



More on Server Crashes, part 1 



f 




5-74. 



* 199Q, 1991 Open Softwae Boundatiaa 



5-150 



Module 5 — File Systems 



Student Notes: More on Server Crashes, part 1 

Here moe, larry, and cuiiy arc the names of NFS servers. Each contains a file system that has been mounted, 
respectively, in InfsIA, IrtfsIB, and IrtfsIC (i.e., the client has set up its directory hierarchy so that all NFS mounts 
are in one directory). Suppose one's current directory is in the root directory of curly 's file system and one 
executes thepwd command. The result should be IrtfslC. How does the pwd command work? It determines the 
inode of the current directory (".'*). and then searches the parent directory ("..") until it finds the component name 
associated with the matching inode number. It then repeats this procedure backwards along the path until it 
reaches the root directory. 

However, when a mount point is encountered, the parent directory of the mount point does not contain the inode 
number of the root directory of the mounted file system. Instead, the pwd command must issue the Stat system 
call for each entry of the parent directory until it finds the entry that refers to the mounted file system. 

Back to our example. Suppose that NFS server moe is down. When thepw<i command is executed starting with 
curly 's root directory, it will be necessary to Stat each of the entries in the Irtfs directory to determine which of 
them refers to curly. But, since moe is down, the Stat call will hang when it is applied to IrfslA. Thus it will be 
impossible to complete t he vwd c ommand until machine moe comes back up, even though there is no logical 
connection between the path Ir^lC and the machine moe. 

This is especially annoying because both csh and ksh perform ?ipfwd when starting up. 
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5-75. NFS File System 



More on Server Crashes, part 2 
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Student Notes: More on Server Crashes, part 2 

This picture illustrates a safer NFS mount technique. An extra level of directories has been added so as to avoid 
the problems vnthpwd. 
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5-76. NFS File System 



The Problem of (Non)Idempotency 
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Student Notes: The Problem of (Non)Ideinpotency 

As previously mentioned, NFS is typically implemented on top of an unreliable protocol and thus must implement 
reliability guarantees itself. To accomplish this, it exploits the request/response nature of its interaction: if a client 
receives no response to its request, it assumes that the request was lost and repeats it. However, difficulties can 
occur if it was the response that was lost, not the request. 

This situation should be no problem as long as the requests are idempotenu as was discussed on page 5-129. 
Certain requests, however, are known to be nonidempotent. For example, suppose that a remove file request is 
repeated because the first response was lost. The response to the second request indicates an error because the file 
no longer exists. But, other than the error, the desired effect has been achieved — ^the file has been removed, 
though the programmer may end up somewhat confused. 

Wi\h some cooperation by the server, this sort of nonidempotency, known as nondestructive nonidempotency, can 
be made transparent. In the original reference port for NFS, the server maintains a queue of completed 
nonidempotent requests and their responses. If a nonidempotent request fails, the server checks this queue to see 
if this is a repeat of an earlier request (the RPC headers contain a transmission id (xid) to facilitate this duplicate 
detection). If it is, then the server repeats the previous response. 

However, as the next slide shows, there are other, more subfle cases that are not dealt with. 
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5-77. NFS File System 



A Problem Case* 
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Student Notes: A Problem Case 

A side effect of a creat request is to truncate the file to zero length if it already exists. In this example, the 
intention was to tnincate the file and then write to it, but the result was the opposite: the file was written to, then 
truncated. The problem is that, though the write request and the creat request are by themselves idempotent, more 
complicated interactions have occuned. That is, idempotency itself is not sufficient 
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5-78. NFS File System 



Fixing the Prob 
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Student Notes: Fixing the Problem 

OSF/l solves this problem by using the technique described in the paper referenced on the previous slide. The 
NFS server maintains a cache of active and completed requests. Items stay in this cache for a finite period (2 
seconds). When the server receives a request, it immediately checks if it is a duplicate of a request still in the 
cache. If it is, and if the original is stUl in progress, then the duplicate is discarded, i.e. the client timed out 
prematurely. If the original completed successfully, the duplicate is again discarded. (Here we are assuming that 
the response was not lost but that the client again timed out prematurely — ^finom observation, this is the usual case.) 

If the response was indeed lost, the client will continue to retry the request; eventually the original request will 
have been removed from the cache, so that a retry will not be recognized as such and will actually be retried. The 
problem outlined in the previous slide will not occur, since the client does not move on to its next request until it 
finally gets a response from its current request. If the original failed, the server retries the duplicate (there is no 
particular rationale for retrying the duplicate other than that this is the behavior of the original implementation of 
NFS). 



5-159 



Module 5 — File Systems 



5-79. NFS File System 



Optimizing NFS Writes in OSF/1 
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student Notes: Optimizing NFS Writes in OSF/1 

Nonnally (i.e., when using a local file system), when one writes to a file a span of data that does not fill an entire 
buffer from the cache, the block I/O subsystem first reads a whole block, then modifies the desired portion of the 
block. To eliminate the need for these (expensive) reads when writing to an NFS file, the block I/O subsystem 
keeps track of what portion of a buffer has been modified (using two new fields in the buf structure: bjUrtyqff 
and b_dirtyend). Thus when the buffer is "cleaned," just the modified portion is written to the server. 

This presents a problem if the entire buffer is read by an application before the modified portion is sent to the 
server consider a situation in which bytes 2048 through 8191 of a file are modified on the client, and no blocks of 
the file currently reside in the client's cache. An 8K buffer is allocated on the client, but only locations 2048 
through 8191 are written. At this point, a thread on the client attempts to read the entire 8K portion of the file. 
Rather than complicate the client-side code so that it will recognize that it must first fetch bytes through 2047 
from the server, the client, whenever it reads from an NFS file that it has recently written to, first cleans its buffer 
cache of blocks from this file (by sending dirty blocks to the server). Then it checks the attributes of the file with 
the server and fetches the block from the server if necessary. 

For further discussion about this implementation of NFS, see Macklem, 1991. 
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5-80. NFS File System 
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Student Notes: Duplicate Detection 

As mentioned previously, duplicate detection relies on an x/^ supplied by th^ RPC level . However, 
retransmissions arc perfonned by a pair of nested loops. In the inner loop, retransmissions are done by the RPC 
layer, which does not modify the value of the xid. However, after this loop is perfonned a finite number of times, 
control passes to the outer loop, which is performed by the NFS layer (client side). In the original reference port, 
at each iteration of this loop the xid would change. OSF/1 (which does not have separate NFS and RPC layers, 
but combines them mto a single layer) fixes this by ensuring that the xid never changes during retransmissions. 
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5-81. NFS File System 
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Student Notes: Parallelization of NFS 

At the higher levels, the parallelization is very similar to that of UFS: there are blocking RW locks on rtfsnodes to 
protect access to files at the level of operation described by uio structures. Simple locks are used to synchronize 
updates to vnodes and rfsnodes. Reads of these data structures need only be protected on architectures with 
"bogus memory." 

Simple locks are used at lower levels for synchronization of NFS data structures. For example, the client side 
maintains a queue of NFS RPC requests (which are waiting for responses). The server side maintains a table of 
active and completed requests, accessed via a hash table. 
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Exercises: 



1. Explain the use of the reference count in the system file table entries. 

2. a. What are the roles of the yfsops and vnodeops data structures in the abstraction of the file system 

concept? 

b. Some versions of UNIX maintain a '*mount table" in the kernel, representing in tabular fonn which file 
systems are moimted where. How is this sort of information represented in OSF/1? 

c. Why is it necessary for a thread to hold a lock over the entire period during which it is using the offset 
field of the system file table entry? 

3. a. How does the buffer cache facilitate concurrent I/O and computation? 

b. Why are blocks in the buffer cache identified by vnode and block number? 

c. What happens when two threads simultaneously access a file block that is not currently in the cache? 

4. a. How are directories protected fiom concurrent conflicting updates? 

b. Explain the concept and use of capabilities in the directory lookup cache. 

5. a. What aspects of the operating system limit the maximum possible size of the file? 
b. What are the perfoimance problems inherent in the standard S5 file system? 

6. a. How is free space represented in a UFS directory? 

b. List the techniques used in the UFS file system to improve performance. 

c. What are the two different policies for allocating fragments for a file? Under what circumstances is each 
policy used? 

d. Suppose that two threads arc extending the size of two different files within the same file system. What 
data structures need to be protected from concurrent access? What types of locks are employed for this 
protection? Under what circumstances can the two threads proceed without one having to wait for the 
other? 

7. a. List three differences between NFS semantics and UNIX semantics. 

b. Explain how generation numbers are used. 

c. What is the function of nfsbiod processes? 

d. Why is the server's buffer cache accessed in a synchronous write-through fashion? 

e. What are the differences between hard mounts, soft mounts, and interruptible hard mounts? 



5-166 



Module 5 — File Systems 



f. Why is it necessary for the server to maintain a queue of recent NFS requests? 
Advanced Questions: 

8. A thread executing an I/O system call involving an ordinary file must obtain blocking locks on the file table 
entry and on bikers in the buffer cache. On a multiprocessor, a simple lock is needed for operations on the 
vnode. Why arc all of these locks necessary? Could fewer be used? 

9. On page 5-77, we discuss the use of timestamps to avoid a race condition. Why isn't this race condition dealt 
with by combining the lookup and delete operations into a single operation? 
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Module Contents 

1. special Files 6-4 

Identifying devices and drivers 

Flow of control 
Data structures 

2. Dynamic Configuration 6-10 

3. Device Drivers 6-16 

Summary of the interface 

4. Terminal I/O 6-30 

Line disciplines 

POSIX session management 
Data structures 
Flow of control 
Pseudo terminals 

Module Objectives 

In order to demonstrate an understanding of device drivers and terminal I/O, the student should be able to: 

• explain the problem of aliasing in special files and how it is dealt with in OSF/1 

• list the steps necessary to dynamically add a module to the operating system 

• list what has been done in the OSF/1 kernel to support intemationalization 

• list the data structures supporting terminals and sessions 
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6-1. The Big Picture 
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Student Notes: Device Drivers 

OSF/l does not provide any device drivers itself, since they are necessarily extremely machine-dependent. 
Device drivers are, however, supplied with the reference ports, and they can be used as a guide for constructing 
one's own device drivers. OSF has added th e dynamic configurability and loadin gofdevice drivers. In 
particular, one of the reference ports is for a symmetric multiprocessor, and its device drivers provide an example 
of how other device drivers may be parallelized. 

This material is discussed in chapters 17 and 18 of Open Software Foundation, 1990a. 
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6-2. Special Files 



Devices 



Accessed wisi special files_ 

— jlock interface /^^^ ^4^^ 

— character interface ^^ /u/A:^ c^cA^ 

Identified by rfmce /t«mi>er A r 

— major portion identifies jiriver - ^/^ff 

— minor portion interpreted by driver ^/r h 



6-2. © 1990, 1991 Open Software f^Nindation 



6-4 



Module 6 — Device Drivers and Terminal I/O 



Student Notes: Devices 

Devices are treated as a special form of a file in that they arc named by paths in the directory hierarchy. A device 
may be accessed via two different interfaces: the block interface, meaning that all access is through the buffer 
cache, and the character interface, meaning that the buffer cache is not used. 

A device is identified by a device nimiber that has two parts: the major portion, identifying the driver, and a minor 
portion to be interpreted by the driver but usually identifying the device, among other things. 
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6-3. Special Files 



Device I/O: Flovi^ of Control 
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Student Notes: Device I/O: Flow of Control 

special files arc represented by inodes in both the S5 and UFS file systems. However, the vnode set up for the 
inode refers to the vnode operations for special files. Thus, for example, a write system call results in a caU to 
specjvrite. 

These special file vnode operations must identify the device driver that controls the device. They do so by using 
the major portion of the device number as an index into the cdevsw table for the character interface and the 
bdevsw table for the block interface. (However, for the block interface, the driver is actually called from the block 
I/O routines.) Each entry in the cdevsw and bdevsw tables is a structure containing entry points of the associated 
driver. 
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6-4. Special Files 



Aliases and Shadows 
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Student Notes: Aliases and Shadows 

A difficulty with devices is that one device might have multiple names {special files). This arrangement could 
lead to problems: for example, when a device driver's close routine is called, the driver must be assured that this 
is the last close of the device, regardless of the name used to open it. For accesses to a device via its block 
interface, we must ensure that, no matter which name of the device is used, all accesses use the same buffers in the 
cache. 

All vnodes (representing open files) of the same device arc linked on a chain headed by a specalias structure. The 
actual links are contained in specinfo structures, which are allocated along with the vnodes when the underlying 
inodes arc brougjit into primary memory. 

Another problem occurs when one block device has multiple names. Since blocks in the buffer cache are 
identified by the pair of vnode address (of the block device) and block number, there would be multiple names for 
each block of the device, one for each of the device's path names. Thus, depending upon which special file is 
used to access a particular block, the block would be identified differently in the cache. This problem is avoided 
though the use of shadow vnodes. 

ff a device is opened via its block interface, then the system allocates a shadow vnode. If the same device is 
subsequently opened via its block interface but with a different name, then the same shadow vnode is used. This 
vnode is used to refer to the device in all accesses to the buffer cache, thus ensuring consistency. 

A related problem might occur when a mounted file system is accessed via its block special interface. In this 
situation, a single block might have two identities in the cache; it is a block within an ordinary file and it is a block 
within the block device. This problem is dealt with by prohibiting access to a mounted file system via its block 
device interface. 
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6-5. Dynamic Configuration 



System Configuration 



• Boot-time activation of driver (BSD) yO^ 

• Dynamic loading and configuring of subsystems (OSF/1) 
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Student Notes: System Configuration 

In BSD-style autoconfiguration, device drivers are statically linked into the kernel. At boot time, 
autoconfiguration code determines which devices are present and "activa^^e appropriate drivers. 



OSF/1 supports dynamic configuration of: 

• device drivers 

• file systems 

• streams modules and drivers 

• networic protocols 

Drivers and other modules may be loaded into or unloaded from a nmning system. 
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6-6. Dynamic Configuration 



Dynamically Adding a Driver 



file system 



driver ; 



6-6. 



configure daemon script 



load driver 

configurc(...) 

mknod(...) 



kernel 




I 
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Student Notes: Dynamically Adding a Driver 

A driver is loaded into the operating system with the aid of the run-time loader, as wiU be discussed in Module 10. 
The run-time loader links the driver to the rest of the operating system, but the loaded driver is responsible for 
linking the rest of the operating system to itself. 

Each dynamically configurable driver has a cor^gure entry point that is called after it has been loaded. The 
bottom half of the driver, i.e. that portion of the driver that responds to interrupts , must link its interrupt handler 
into the rest of the kemel. It accomplishes this linkage by calling a pair of routines: 

• handler add registers a new interrupt handler 

• handler enable "turns on'* a registered interrupt handler 

Th efop/ifl/f ofthe driver, i.e. that part of the driver called in the thread context in response to system calls, must 
make itself known to the rest of the kemel. It does this by creating entries in one or both of the cdevsw and the 
bdevsw by calling: 

• cdevsw _add 

• bdevsw_add 

In both cases, the caller either supplies a major device number or is assigned one. 

For further information on dynamic configuration, see chapters 1 through 6 in Open Software Foundation, 1990a. 
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6-7. Dynamic Configuration 



Configuring the Interrupt Handler 



m^scrspt 



device 



^ 



itable 




handlers 



6-7. 



^(^^ 



IC 



'J 



' 1990, 1991 Open Software Foundttioo 



6-14 



Module 6 — Device Drivers and Terminal I/O 



Student Notes: Configuring the Interrupt Handler 

In the BSD kernel, the notion of interrupt vectoring is "wired into" the kernel; i.e., there is no convenient 
technique for adding intemipt handlers dynamically. OSF/1 provides an approach for adding and removing 
interrupt handlers dynamically, though the method involves some very machine-dependent facilities and must be 
tailored for each architecture. 

When an interrupt occurs, an interrupt dispatcher is invoked in some machine-dependent fashion. The dispatcher, 
using machine-dependent techniques, consults the itable for the appropriate handler and forwards the interrupt to 
it (i.e. calls it). 

A typical itable might be an array indexed by the interrupt level, as shown in the picture. Each element of the 
array would head a linked list of handlers for interrupts at that level. Chosing the correct handler would depend 
upon machine-dependent information, such as a vector address. 
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6-8. Device Drivers 



Major 


Driver Entry Points 


corfigure 


called after a driver has been dynamically loaded to link itself into the rest 
of the kernel 


probe 


called at boot time to determine if the device is present; used with 
BSD-style autoconfiguration 


attach 


caUed at boot time after it is known that the device exists to perfonn device 
initialization; used with BSD-style autoconfiguration 


open 


called on every open of a special file (device) 


close 


called on the last close of a special file 


read, write 


called to initiate transfers for character special files 


strategy 


called to initiate transfers represented by ftw/ structures 


ioctl 


called to handle miscellaneous requests to the driver 


intr 


called in response to interrupts 
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Student Notes: Major Driver Entry Points 

The probe and attach routines arc called only at boot time and only with BSD-style autoconfiguration. If they are 
used, then cortfigure is not needed, and vice versa. 



6-17 



Module 6 — Device Drivers and Terminal I/O 



6-9. Device Drivers 



Driver Entry Points: Open 



open(dcw, flag, type) 
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Student Notes: Driver Entry Points: Open 

• dev: device number (major and minor) 

• flag: flags parameter from the open system call 
— 0_RDONLY,0_RDWR.O_NDELAY,etc. 

• type: indicates whether the character or block interface is being used (this argument is new and rarely used) 
Possible actions: 

• initialize per-device state infomiation 

• "turn on" device 

• wait for device to be ready (e.g. wait for carrier-detect) 

• check device status (e.g. opening for write, but no write ring on a tape drive) 

• etc. 

Requirements: return if everything is ok, error code otherwise (e.g. ENXIO if device does not exist, EBUSY if 
device must be used exclusively but is busy) 
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6-10. Device Drivers 



Driver Entry Points: Close 



close(6ev, flag, type) 
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Student Notes: Driver Entry Points: Close 

• dev: device number (major and minor) 

• flag: flags parameter from the open system call 

• type: indicates whether the character or block interface is being used (this alignment is new and rarely used) 
Possible actions: 

• turn device "off 

• hang up phone line 

• etc. 
Requirements: none 
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6-11. Device Drivers 



Driver Entry Points: Read/Write 



readidsv, uio), write{dev, uio) 
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Student Notes: Driver Entry Points: Read/Write 

• dev: device number (major and minor) 

• uio: pointer to the uio structure describing the request 
Actions: 

• does I/O directly in simplest drivers 

• for terminal I/O drivers: calls line discipline transfer routine 

• for drivers which transfer directly into or out of the buffer provided by the usen calls physio (which fetches 
and wires the user's buffer into primary memory), which then calls strategy 

Requirements: 

• if this routine is doing the transfer, it should set uio->uio_resid to the number of bytes not transferred 

• returns if no errors, or returns an error code (e.g. EIO for an I/O em)r, EFAULT if an invalid address was 
given for a buffer) 
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6-12. Device Drivers 



Driver Entry Points: Strategy 



strategyQyp) 
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Student Notes: Driver Entry Points: Strategy 

• bp: pointer to the bi^ stmctarc describing the request 
— either initiates the I/O lequest or queues it for eventual action 
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6-13. Device Drivers 



Driver Entry Points: loctl 



ioctlidcvy cmd, data, flag) 
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Student Notes: Driver Entry Points: loctl 

• dev: device number (major and minor) 

• cmd: command code (second ai^gument of the iocti system call) 

— encoded in this conmiand code is a description of the data that needs to be transferred from user to system 
or from system to user, this allows the hi^er-level code to perform this transfer for all drivers 

• data: pointer to the ai^gument (either in or out) of the loctl; these are interpreted differently by each driver 

• flag: flags parameter from the open call 
Possible actions: 

• turn terminal modes 

• rewind a tape drive 

• etc. 

Requirements: return an appropriate error code (usually ENOTTY if the command makes no sense) 
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6-14. Device Drivers 



Driver Entry Points: Interrupt 



intr(6ev) 
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Student Notes: Driver Entry Points: Interrupt 

• dev: the hardware device number of the interrupting device, not its major and minor device numbers 
Possible actions: 

• check for and react to errors 

• wake up threads waiting for the I/O completion 

• acknowledge the interrupt in the controller registers 
Requirements: 

• be quick! 

• don't sleep; execution is in the interrupt context and not in the context of any thread 
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6-15. Terminal I/O 



Terminals 



6-15. 
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• Line discipline technology ~L^0 j ^^ — 



• Internationalization 

• Session control 
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Student Notes: Terminals 
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6-16. Terminal I/O 



Internationalization 



• 8-bit clean 

• Shift-JIS support for Asian character sets 
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Student Notes: Internationalization 

Earlier versions of UNIX were designed strictiy for use with the ASQI character set This is a seven-bit character 
set and, programmers being programmers, the extra eighth bit was used for a variety of purposes. The 
seven-bit/eight-bit problem has been cleaned up in OSF/1, and the entire kernel and all of the libraries are 
eight-bit clean: no special use is made of the eighth bit in characters, and thus eight-bit character sets can be 
supported. 

A much more difficult problem is dealing with character sets in which characters are larger than bytes. OSF/1 
includes support for Shift- JIS, but this is not considered the final word on the subject and more will be available in 
Release 1.1. 
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6-17. Terminal I/O 



Sessions 



$^$^on 



pn)cess group B 



process group A 



'-N 



process group D 




6-17. 



' 1990. 1991 Open Software PDnndatian 



6-34 



Module 6 — Device Drivers and Terminal I/O 



Student Notes: Sessions 

POSIX introduced the concept of sessions to clean up the BSD notion of job control. A session is a terminal 
session, and hence a collection of processes sharing a terminal. Traditionally there has been the notion of 
foreground processes and background processes: foreground processes are affected by signals generated by key 
strokes, background processes are not. Job contro l is a means for moving processes back and forth between the 
jDreground and the background. 

The picture shows three process groups whose names are the process id of their first and founding members. 

• Process A formed the session and spawned processes B, C, and D 

— process A is in its own process group 

— process B fomied a new process group with itself as leader (giving its name to the group) and C as another 
member 

— process D formed another new process group and spawned two children that stayed in the group 

• Each process group forms a "job" that can be suspended or placed in the foreground or background 

• A stop signal suspends a process group 

— sent by a thread in another process (usually an ancestor) 

— sent by the kernel due to actions on the terminal (e.g., a background process reading from a terminal) 

— all threads within each process of the process group are suspended (using task_suspend) 

• A continue signal (usually from an ancestor) resumes a stopped process 

— if an orphaned process receives a stop signal because of actions on the terminal (if, say, it is a background 
process and is reading from the terminal, and thus receives the SIGTTIN signal), it is unlikely that any 
thread will send it a continue signal; such a stop signal should be ignored (the I/O system call will return the 
error EIO). Note that this differs from BSD's solution — ^in BSD the process would have been sent the 
SIGKILL signal (and hence would have been terminated) 

— the generalization of the orphaned process is the orphaned process group: a process group whose members 
have no parents within the session 
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6-18. Terminal I/O 



Orphaned Process Groups 



r 



Session 
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process group A 



process group 



process group D 
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Student Notes: Orphaned Process Groups 

• If process D temiinates, then the init process adopts processes E and F 

— process group D is thus an orphan, since it has no processes with a parent in the session 

— processes E and F receive the EIO error if they attempt to read from their terminal 

A process group is considered an orphan if none of its members have an ancestor that is the session leader. Thus 
in the picture, if process C is moved to process group D, then process group D is no longer an orphan. 
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6-19. Terminal I/O 



Terminal Data Structures 



number of process groops 



controlling tenninal 



hashM 



josmhEO. 



sessicwi 



Km4 



Jobcoaalificarion 



leader 



session structure 



pgrp structure 



11 



^12. 



proc structure 



ME. 




vnode structure 



character queues 



state infcHmation 



line disc^line 



foregioundpgrp 



tty Structure 



proc structure 



J^ 



JUL 



proc structure 



hash link 



members 



jSSSiSSL 



peiptf 



iobc Qualification 



pgrp structure 
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Student Notes: Terminal Data Structures 

The jobc qualffication field of Haupgrp structure is used to indicate whether or not the process group is an orphan 
(and hence not qualified to receive job control). The field contains a count of the number of processes in the 
process group whose parents are both outside of the process group and qualified for job control. 
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6-20. Terminal I/O 



Line Discipline 
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Student Notes: Line Discipline 

This slide illustrates how the device number (fiiom the vnode structure) is used to identify the tty structure and 
hence to identify the entry points into the terminal's line discipline. Each terminal device driver maintains a table 
mapping minor device numbers into pointers to tty strucmres. 



6-41 



Module 6 — Device Drivers and Terminal I/O 



6-21. Terminal I/O 



Terminal I/O Flow 

k^board 



iffl^W 



user program 
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Student Notes: Terminal I/O Flow 

When a character is typed on the keyboard, the driver's read interrupt routine (rint) is called (in the intemipt 
context). It detennines which line discipline is being used by consulting the tty data structure associated with the 
tenninal. In this example we assume that the standard tty line discipline is being used. Thus control is passed to 
the ttyinput routine. This routine looks at the tty structure to detennine the terminal's mode. If it is in cooked 
mode, then the incoming characters are interpreted as parts of lines of text which may be edited. Characters are 
first placed on the raw queue, where they may be edited in response to edit characters. Once a line delimiter (e.g., 
carriage return) is received, the line of text is copied to the can (canonical) queue. Otherwise, if the tenninal is in 
either raw mode or cbreak mode, incoming characters cannot be edited, and are left on the raw queue. If echoing 
is enabled, then the ttyinput routine calls the ttyoutput routine to echo the characters. Since this processing occurs 
in the interrupt context, the interrupted context is resumed after ttyinput returns. 

In the context of a thread performing a read system call, control enters the device driver's read routine which 
consults the tty structure and, in our case, calls the tty line discipline's ttread routine. This routine detennines the 
mode of the terminal. If the terminal is in cooked mode, ttread looks in the can queue for characters. Otherwise it 
looks in the raw queue. If there are no characters in the appropriate queue and if blocking is permitted, the calling 
thread blocks until charaaers are received. If characters are available then they are copied to the user's buffer. 

In the context of a thread perfonning a write system call, control enters the device driver's write routine, which, in 
our case, calls the line discipline's ttywrite routine. This routine copies characters into the kernel from the user's 
buffer. Those characters requiring no special processing are appended to the out queue. Characters requiring 
special processing are processed by the ttyoutput routine (the primary purpose of this routine is to deal with 
characters not present in the terminal's character set; this routine is rarely needed in modem systems). 

The device driver, usually executing in the interrupt context, transfers characters from the out queue to the device. 
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6-22. Terminal I/O 



Terminal I/O Data Structures 
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Student Notes: Terminal I/O Data Structures 

The raw queue, can queue, and out queue of the previous slide are instances of data structures known generically 
as clists, which represent queues of 8-bit characters. The format of the list, shown in the accompanying picture, 
consists of a header followed by a number of fixed-size cblocks. The cblocks are typically 64 bytes in length, 
with 12 bytes of overhead and 52 bytes of data (cjnfo). 

Of particular interest is the cjjuote, which is used to indicate which characters within cjrfo have been "quoted," 
meaning that they are not to be interpreted as providing any sort of special function such as erasing a character or 
suspending the process group. In earlier versions of UNIX, such quoting was done by setting the eighth bit of a 
character. However, this technique only works with English character sets. 
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6-23. Terminal I/O 



Pseudo Terminals 



machine A ^M 



.•:-:-y*M**MiiMiiMtM*|:-:->:v 

m ilogin m 







communication line 



machine B 



^ifmmfminmttmfimmmmiiMmmimmmm 



-• j application I 



machine A 

i riogin !<*■ 




commimication line 



machine B 
■^ l riogind I I applicationl 

^line discipline j 



I 



pseudo tty 
control ! />slave 



6-23. 



1990, 1991 9pen SoBwaie Rnmdttioii 



/^^f A' ^ /«-5#- ' *!^lt 



6-46 



Module 6 — Device Drivers and Terminal I/O 



Student Notes: Pseudo Terminals 

Pseudo tenninals are used to present a teiminal interface to an application that is not connected to a real terminal. 
As an example, consider a remote-login-type application. In the top poition of the picture, the real tenninal is 
connected to machine A, and the user is running rlogin so as to run an application on machine B. This application 
might be an editor, which might attempt to make changes to the terminjd's mode, e.g., switch from cooked mode 
to cbreak mode. However, such attempts would fail because it is actually connected to a communication line that 
is different from a terminal and does not respond to requests to change modes. 

One can think of two approaches to this problem. The first would be to forward the terminal-oriented system caUs 
to the machine to which the tenninal is connected. The other would be to run the line-discipline code on the 
application's machine (machine B in this example). Berkeley UNIX, and hence OSF/1, use the latter approach. 

A pseudo device driver or a pseudo terminal is set up in machine B 's kernel. It appears to the rest of the operating 
system to be a pair of ordinary device drivers, but the pair of drivers commimicate with each other instead of with 
devices. A user process, in this case the rlogind, communicates with rlogin via the communication line. 
Incoming bytes are written to one of the pseudo devices — the control device. Output through this device is made 
to reappear as input from the slave device, i.e. as if a character had just been received from a terminal. This 
character is then processed by the line discipline code discussed on page 6-43: characters arc queued on either the 
raw queue or the can queue. When the application issues a read system call, it receives these characters after 
they have been processed by the line discipline. Output from the application is processed by the line discipline as 
if the characters were being sent to a real terminal, but instead they are made to appear as incoming characters in 
the control pseudo device. These characters are read by the rlogind, which sends them across the communication 
line to rlogin. 
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Exercises: 



1 . a. Is there any difference between a special file whose inode is in an S5 file system and one whose inode is 

in a UFS file system? 

b. Why are the specalias and specinfo stmctures necessary? 

2. a. List the steps necessary to add a module to the operating system dynamically. 

b. Why is it necessary that dynamically loadable modules have configure routines (for example, why 
couldn't the work performed by the configure routine be perfonned by user-level code?)? 

3. What arc the differences between BSD device drivers and OSF/1 device drivers? 

4. a. What is an "orphaned" process group? 

b. How is it determined what a terminal's line discipline is? 

c. Explain the relationship between pseudo terminals and device drivers. 
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Module Contents 

1. Streams Concepts 7-4 

Streams components 

Pipelines 
Multiplexers 

2. Message Flow 7-24 

Messages 

Standard entry points 
Flow of control 

3. Implementation of Streams 7-38 

Standard data stmctures 

OSF/1 -specific data strucmres *^ 

Representing an open stream 

Qoning 

4. Parallelization 7-50 

Synchronization options 

Data stmctures 
Flow of control 

Module Objectives 

In order to demonstrate an understanding of the concept of streams and tiieir implementation in OSF/1, the student 
should be able to: 

• list the three types of streams components 

• list and show the interconnections between the data structures used to represent a streams message 

• explain why it is necessary for streams-oriented service calls to be serialized 

• list the data stmctures used to represent and access an open stream 

• explain the purpose of cloning 

• list the synchronization options available in the OSF/1 implementation of streams 
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7-1. The Big Picture 
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Student Notes: Streams 

The bes t general ing oduction to streams is AT &T. 1989. The OSF/1 implementation of streams is discussed in 
chapter 13 of Open Software Foundation, 1990a. 
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7-2. Streams Concepts 



Streams 



Kernel analog of the shell concept of a pipeline 

OSF/1 streams arc a reimplementation of SVR3 streams, with the 
important addition of parallelization 
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Student Notes: Streams 

A shell pipeline is a unidirectional stream of bytes processed by one or more filters. Akej:B£Lstj:can is a 
bidirectional stream of messages being processed in one or more modules. The endpoints of a kernel stream can 
be in two user processes, or, most commonly, one endpoint may be in a user process and the other in a device 
driver. 

In shell pipelines, each filter is implemented as a separate process. This technique could be extended to kernel 
streams through the use of kemel threads. However, the original designer of streams, Dennis Ritchie, felt that so 
many streams modules would be active at once that, even with very lightweight kemel threads, a thread per 
module would be too expensive. Thus each module is a collection of procedures that can be called in a variety of 
contexts. Associated with each module are data structures to contain its state, so that the module's execution can 
be started in the context of one caller and continued in the context of another. 
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7-3. Streams Concepts 



Stream Components, part 1 
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Student Notes: Stream Components, part 1 

Each module consists of a set of routines to process data and (possibly) a pair of queues, one for each direction. 

A simple example of a module is one that capitalizes all characters going in either direction. A more complicated 
module might be one that encrypts data going downstream and decrypts data going upstream. 
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7-4. Streams Concepts 



Stream Components, part 2 






Stream head *^^ ^/^^^s- 
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Student Notes: Stream Components, part 2 

Stream head is a special case of a stream mcxiule: it supplies the interface to the system-call layer, converts system 
calls into messages sent down a stream, and converts messages arriving from the stream into responses to system 
calls. 
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7-5. Streams Concepts 



Stream Components, part 3 



^ K^:*tl$^\^.*4 dnver V.^^-'^ ^ :^^^V 



•»: ■" 'T^t -^ N'^'-o 



iMtlMtlrHUMiiMHHllttMlllllMlMllltliilU 



7-5. 



' 1990, 1991 Open Softwue Fcnmdatioa 



7-10 



Module 7 — Streams 



Student Notes: Stream Components, part 3 

A streams driver can be an interface to a real device or can be an interface to other streams, as will be seen. 
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7-6. Streams Concepts 



Stream Setup 



fd = open(7dev/stream_devlce", 0_RDWR); 
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Module 7 — Streams 



student Notes: Stream Setup 

A stream may be created by opening a streams device. 

Streams devices appear as character special devices, i.e., tliey are identified as special files and have entries in the 
cdevsw. 

A streams device may be read from and written to immediately after being opened, but, at this point, it doesn't 
supply much additional functionality over non-streams devices. 
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7-7. Streams Concepts 



Stream Push 



ioctl(fd, LPUSH, "module name"); 




o 
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Student Notes: Stream Push 

A module may be inserted at the top (pushed) by executing an iocti with the I_PUSH command. Data being sent 
in either direction through the stream will now be processed by the module. 
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7-8. Streams Concepts 



Linking Streams, part 1 



fdtop = open(7dev/top", 0_RDWR); 
fdbottom = open(7dev/bottom", 0_RDWR); 




o 







>XJ bottom?. ^.^ 
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Student Notes: Linking Streams, part 1 

In this picture the streams driver module top is a multiplexer, i.e., it is an interface between a stream and other 
streams, not an interface between a stream and a device. 



7-17 



Module 7 — Streams 



7-9. Streams Concepts 



Linking Streams, part 2 



ioctl(fdtop, LLINK, fdbottom); 
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Student Notes: Linking Streams, part 2 

The bottom pipeline of the previous picture has been linked beneath the streams driver top. The streams driver 
top now has an upper and a lower part. Part of top's ftmaion will be to transfer data from the upper part to the 
lower part and vice versa. 
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7-10. Streams Concepts 



Multiplexing Streams, part 1 



fdbottom2 = open(7dev/bottom2", 0_RDWR); 
ioctl(fdtop, LLINK, fdbottom2); 



stream head 
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Student Notes: Multiplexing Streams, part 1 

In this picture the driver for top is responsible for the multiplexingAlcmultiplexing of messages coming up from 
below or going down from above. For messages going downwards, code supplied in the top module must decide 
through which lower stream the data should be sent It uses its own criteria for doing so, but is likely to base its 
decision on the contents of the message itself. 

Here top is a one-to-many multiplexer. 
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7-11. Streams Concepts 



Multiplexing Streams, part 2 
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Student Notes: Multiplexing Streams, part 2 

In this example we show a more complicated arrangement of multiplexers. TCP is a many-to-one multiplexer. IP 
in the picture is a one-to-many multiplexer, though in practice it is a many-to-many multiplexer. 

To create this arrangement, first the IP, ether, and token streams are opened, and then the latter two streams are 
linked to IP. Then the TCP stream is opened and the IP stream linked beneath it. Each stream connected to the 
top of the TCP module represents a separate TCP connectioa 

(N.B.: in OSF/1, the TCP/IP protocols are implemented not in the streams framework, but in the socket 
framework. TCP/IP is used here merely as an example.) 



7-23 



Module 7 — Streams 



7-12. Message Flow 
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Student Notes: Stream Message Flow 

Each module defines (usually) a wput routine and an rput routine to be called when a message arrives from above 
(wput) or below (rput). From within a module, the appropriate put procedure of the next module is caUed by 
calling the routine putnext. 

Simple modules modify incoming messages in-place and pass them on to the next module (by cdiJlmg putnext). 
Other modules may need additional resources to process a message, or may need to defer processing. These 
modules have the ability to queue messages. Associated with these queues (one for each direction) is a size limit 
known as the high-water mark. 

Because of the finite capacities of these queues, flow control must be established. A module that is itself capable 
of queueing must check for queue space ahead of it (by calling canput) before sending the message. The canput 
routine looks ahead for the first module that is capable of queueing and returns true if and only if there is room for 
another message in that queue. If there is no room, then the calling module must put the message in its own queue 
(the previous module's canput routine checked that there was room on the queue before it sent the message). A 
module may also defer the processing of a message by placing the message at the end of its queue (in either case, 
enqueuing is accomplished by calling puf^). 

Note that if flow control propagates upwards to the stream head, threads issuing write system calls wiU eventually 
block waiting for space. However, if flow control propagates downwards to a streams driver serving as an 
interface to a real device, then incoming data will be lost since there is no more buffer space and no notion of 
blocking the source of the data. 
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7-13. Message Flow 



Stream Service Procedures 
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Student Notes: Stream Service Procedures 

A module that may defer the processing of a message must have a service procedure that can be called when 
further processing is possible. Processing may be deferred because of: 

• flow control 

• yielding to more important processing 

• shortage of buffer space 

To cause a service procedure to be called sometime in the future, it is enabled, which puts it on a list of enabled 
service procedures. Enabled service procedures are called by members of a special streams thread pool. 

A service procedure is automatically enabled whenever pM^^ is called (and the queue was previously empty) and 
whenever a getq is called that removes sufficient messages from a forward queue that the queue size falls below 
the queue's low-watermark. 
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7-14. Message Flow 



Messages in Streams 



A message as viewed by the designer of a protocol: 



header 



data 



trailer 
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Student Notes: Messages in Streams 

The simplest stream module merely passes its input to the next module in the stream. This transfer is very 
efficient, since the data is represented by a linked list of message blocks and is passed by reference. A message 
consists of a linked list of message blocks, each of which points to a data block, each of which points into a 
variable-length buffer. The justification for the data blocks is that they allow multiple message blocks to refer to 
the same data block and avoid the overhead of copying (the data block contains a reference count). Representing 
a message as a sequence of message blocks makes it easy, for example, to add headers or trailers to messages (and 
to strip tfiem ofO. 

The message shown in the picture would be implemented as a linked list of three submessages, one for the header, 
one for the data, and one for the trailer. Stripping off the header and trailer or appending additional information is 
then very easy. 
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7-15. Message Flow 



Messages 
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Student Notes: Messages 

A message is represented as a linked list of mblks. Each mblk indirccdy refers to a buffer via a dblk. The dblk, as 
is discussed further on page 7-35, contains the reference count and other information about the buffer. The mblk 
contains a pair of pointers pointing directly into the buffer. These pointers allow the easy representation of 
consuming data from the buffer and putting data into the buffer. 
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7-16. Message Flow 



Message Queue 
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Student Notes: Message Queue 

A queue of messages is represented by linking together the first mblks of each message. 



7-33 



Module 7 — Streams 



7-17. Message Flow 



Virtual Copy (Streams Style) 
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Student Notes: Virtual Copy (Streams Style) 

In many situations, it is necessary for a streams module both to pass a reference to data and to retain a reference to 
the same data. For example, the TCP protocol would send data to be transmitted to the IP protocol, but would 
retain a copy of the data just in case it is never acknowledged and thus must be retransmitted. This "virtual copy" 
is implemented with reference counts: each buffer's dblk contains the buffer's reference count. Creating a virtual 
copy of the buffer merely involves incrementing the reference coimt. Freeing a buffer causes its reference count 
to be decremented; if the reference count is reduced to 0, then the storage is actually liberated. 
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7-18. Message Flow 



Types of Messages 




• Ordinary message types: 


• Priority message types: 


M DATA 


— M.PCPROTO 


— M_PROTO 


— M_ERROR 


M lOCTL 


— M_HANGUP 


— M.CTL 


M lOCACK 


— M_BREAK 
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M DELAY 


M FLUSH 
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M SETOPTS 


— M_START 


M SIG 


— M_STOP 
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Student Notes: Types of Messages 

Each message block is assigned a type that stieams modules use to determine what sort of processing is required. 
There are two classes of messages: those that are subject to flow control {ordinary messages) and those that are 
not {priority messages). 

Since priority messages are not subject to flow control, they are forwarded even if subsequent queues are full. 

For example, an iocti system call might be implemented within a streams component. In this case, a message is 
created whose first mblk points to a dblk of type MJOCTL, which points to a buffer containing the command. 
Subsequent mbVcs of this message point to dbUcs of type M_DATA, which point to buffers containing the 
associated data. This message is passed down the pipeline until it reaches a component that recognizes the ioctI 
request This component performs the desired action and sends back a priority message whose first mblk points to 
a dblk of type MJOCACK. Being a priority message, it is passed up to the stream head immediately without 
being queued, and thus it is dealt with at the stream head before any ordinary messages that might be in the 
queues. This prioritized delivery is necessary because the thread performing the ioctI system call does not move 
on to make, for example, read system calls until the ioctI system call completes. 

If no component recognizes the ioctI request, then the last component, a streams driver, sends back a priority 
message of type M_IOCNAK. 
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7-19. Implementation of Streams 



Queue Structure 
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Student Notes: Queue Structure 

Each instance of a streams component, whether stream head, module, or driver, is represented by a pair of queue j 
structures. Each such structure links the component to the next component in the pipeline, points to the associated 
queue of messages, and points to two types of data structures. 

The first type of data structure, the qinit structure, contains information shared by all instances of the component. 
It contains the addresses of the pur and service procedures for either the read or write half of the component and 
points to the module irtfo structure, which gives the default values for various parameters. 

The second type of data structure, pointed to by q_ptr, contains infomiation that is private to each instance of a 
component. The contents of this data structure depend upon the component 
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7-20. Implementation of Streams 
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Student Notes: Stream Head 

The stream-head module provides the interface between system calls and the stream. It is the only portion of 
streams in which threads may block: a thread making a system call must necessarily block until it can return. 

The module is represented by a STH (stream head) structure. It contains various pieces of information about the 
stream and refers to the stream-head queue data stmctures. 

Multi ple threads may ac c ess the stre aniMConcurrently. Three types of system calls involve messages and streams: 
loctis, reads and getmigs, and writes and putmsgs. A user thread (executing a streams system call) creates an 
operating systemrequest (OSR) structure to represent the system call. If the request cannot be satisfied 
immediately, then the OSR is queued on one of the STH's OSRQs (one queue for each system call type). 
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7-21. Implementation of Streams 



Representing an Open Stream 
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Student Notes: Representing an Open Stream 

An open stream appears to the rest of the operating system as if it were a character special file. The cdevsw entry 
contains not the addresses of the streams driver's entry points but the entry points of the more general stream-head 
routines. This is important since the interface to the system-call layer is at the stream head and not at the streams 
driver. The stream-head code itself finds the desired stream by accessing the STH structure. 
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7-22. Implementation of Streams 



Device Module Switch Table 
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Student Notes: Device Module Switch Table 

The device module switch table is an array of dmodsw structures, one for each type of streams driver. It is 
accessed by the major portion of the device number and refers to the device's streamtab structure. This structure 
contains the addresses of the qimt structures for each portion of the device (the device may be a multiplexer, so 
the streamtab structure refers to the qinit structures for the lower-side queues as weU). 

Each dmodsw entry also refers to an STHT (stream header table) structure, which is an array of STH structures, 
one per minor device. The STH structure, as we have seen, represents the stream head of a particular stream and 
is allocated when the stream (i.e. minor device) is opened. 

Each stream appears to the rest of the operating system as if it were a character-special device. Thus associated 
with each streams device is an entry in the cdevsw table. However, this entry refers not to the entry points of the 
streams driver, but to the general stream-head entry points. Additional information is needed to find the entry 
points of the streams device itself and to find the STH structure identifying the specific instance of the stream. 
This information is obtained fix)m the dmodsw structure as we have just seen. The major portion of the device 
number is used twice: once to index the cdevsw table to determine that this is a stream, and again to index the 
dmodsw table to find the streamtab data, structure and the STHT. Rnally, the minor portion of the device number 
is used to index the STHT to find the STH that identifies the actual instance of the stream. 
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7-23. Implementation of Streams 



File Module Switch Table 
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student Notes: File Module Switch Table 

The file module switch table is an array of finodsw structures. It is accessed by the module name and refers to 
each module's streamtab stmcture, which in turn refers to the module's qinit structures. Thus individual streams 
modules are identified through the file module switch table. 
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7-24. Implementation of Streams 



Cloning a Stream 
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Student Notes: Cloning a Stream 

The notion of cloning is an important concept that was introduced in SVR3 for the support of streams and has 
been extended in OSF/1 for use in any character-special file. Qoning is used in situations in which there is a 
varying number of logical devices. 

For example, each connection over a streams implementation of TCP/IP is a logical device; each such logical 
device is represented by a separate minor device. Since the user of the logical device does not really care which 
minor device member is chosen, it is inconvenient to represent each logical device as a separate special file (in the 
/dev directory). Qoning allows one to use just one special file to represent the major device, and to have the 
minor device numbers automatically produced internally in response to opening the major device. 

A clonable device is represented by a special file whose major device number is that of the clone driver and 
whose minor device number is equal to the major device number of the clonable device. Opening this special file 
results in a caU to the open entry point of the clone dnwer, clone^^n, which returns the error code ECLONEME. 
The caller of this routine , .ypec ope n, then calls spec_clone^tnch creates a new vnode, and then calls the open 
routine given in cdevsw as indexed by the original minor device number (i.e., the major device number of the 
clonable device). 

For the case of streams, the call to open results in a call to osropen, which sets up a stream head and tiie device 
driver module, and calls the device driver open routine with the clone flag set. The device driver then finds an 
available minor device number and returns it to the caller and the caller's caller, and so forth, who wiU eventually 
put this in the new vnode. The effect then is that all further I/O requests use the newly created vnode and access 
the newly created stream. 
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7-25. Parallelization 



Parallelization of Streams 
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Student Notes: Parallelization of Streams 

Streams are parallelized in a flilly transparent manner streams modules from an SVR3 system can be put into an 
OSF/1 kernel so as to allow fiilly parallel execution with essentially no changes to the code. Certainly some 
synchronization is necessary. This synchronization is implemented within the standard routines that are called for 
communication between streams and modules. The finest degree of parallelization is at the individual-queue level 
(SQLVL_QUEUE). Associated with the queue is a lock; only one thread may execute within the queue at a time. 
One may als o req uest s ynchronization across a pair of queues w ithin a module (SQLVL_QUEUEPAIR), _across all 
jnstances of a module (SQLVL_MODULE), and across a group of modules (SQLVL_ELSEWHERE). For 
debugging purposes, one may request a single lock for the entire streams system, i.e. only one thread at a time 
may be executing anyplace (SQLVL.GLOB AL).- // >^^ SfP^f- ' ~ ^ 

We first look at queue-level synchronization. Whenever a module is entered (e.g. via. putnext), a check is made to 
see if the queue is locked. If so, then instead of waiting for the queue to be unlocked, the request is queued on a 
synchronization queue, and the call retums immediately. Thus while a thread operates within a module, further 
requests to enter that module queue up on the synchronization queue. When the thread leaves the module, it must 
check if there are any requests in the synchronization queue and then handle each of these requests as if it had 
made them itself. 

The synchronization queue is headed by an SQH data structure, which, for queue-level synchronization 
(SQLVL_QUEUE), is in the queue data structure. This contains a pair of simple locks, one that is the lock on the 
entire queue, and the other that is the lock for operations on the synchronization queue. Each request in the queue 
is represented by an SQ data structure, which refers to the queue being accessed, the routine being called, and the 
message being transmitted. The SQ data structure itself is allocated within the mblk data structure. 

The technique for single-queue parallelization can be extended for coarser parallelization. To achieve queue-pair 
synchronization (SQLVL_QUEUEPAIR), one effectively merges the synchronization queues of the two individual 
queues. This is accomplished through the use of the sq_parent field of the SQH data structure. This field 
normally points to the SQH data structure itself, but it may point to another SQH data structure. In the latter case, 
the target SQH structure is used instead of the source SQH stmcture. So, for queue-pair parallelization, the 
write-side queue's SQH structure points to that of the read-side queue, and there is one set of blocks and one 
synchronization queue for both queues of the module. 

For module-wide synchronization (SQLVL_MODULE), each queue in each instance of the module refers to an 
SQH strucmrc in the modvle^s finodsw structure (or the dmodsw structure if it is a driver). 
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7-26. Parallelization 
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student Notes: SQLVL^QUEUE 

In this picture queue-level synchronization is used. There is space in each queuej for an SQH structure, which 
heads the queue's synchronization queue. For queue-level synchronization, each of the synchronization queues is 
totally independent of the others. Thus each queue may have no more than one thread active at a time, but there 
may be two active threads within each instance of the streams component. 
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7-27. Parallelization 
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student Notes: SQLVL^QUEUEPAIR 

This picture illustrates queue-pair-level synchronization. The sq_parent field of one queue's SQH structure points 
to the other SQH structure, effectively merging the two synchronization queues. Thus we are assured that only 
one thread can be in either queue at a time. 
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7-28. Parallelization 
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Student Notes: SQLVL_MODULE 

This picture shows module-level synchronization. The sq_parent fields of each queue's SQH structure points to 
the SQH structure stored in i}a&fmodsw structure (modules) or the dmodsw structure (drivers). Thus the 
synchronization queues for all queues within all instances of a module are effectively merged: at most one thread 
can be active in any instance of the module at a time. 
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7-29. Parallelization 



Implementation of Synchronization Queues 




callpiu 
procedure 



yes 




T 



yes 



caU 
csqja teral 



call 
csqjocqidre 



7-29. 



' 1990, 1991 Open Softwue Fmnidation 



7-58 



Module 7 — Streams 



Student Notes: Implementation of Synchronization Queues 

In Older to enter a streams module (for example, as part of a putnext request), a thread encodes a request in the 
message's SQ structure and calls the csqjateral routine. This routine checks the lock on the appropriate 
synchronization queue. If the lock is taken, the routine puts the request on the synchronization queue and retums. 
)Vhen a thread finishes a request inside of the module, as mentioned earlier, it checks the synchronization queue 
and handles any queued requests. 

This arrangement could result in a race condition. The calling thread might determine that the queue is busy, but, 
before it can enqueue an SQ on the synchronisation queue, the thread that owned the streams queue determines 
that the synchronization queue is empty and releases the lock. Thus the SQ wiU soon be enqueued, but no thread 
wiU be available to process it. To deal with this, csqjateral checks the lock after it has enqueued the SQ, and, if 
it has been released, then processes the SQ itself. 

When a user thread (executing in kemel mode in response to a system call) operating within a stream head calls 
putnext to send a message to the first module, it calls csq_acquire instead of csqjateral. This routine blocks 
waiting for the lock. When the thread holding the lock is finished with its operation, instead of processing the 
synchronization queue itself, it hands the work over to the user thread blocked in csq_acquire. 
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Exercises: 

1. a. List the three types of streams components. 

b. What happens when one stream is linked to another stream? 

c. What are the functions of a multiplexer? 

2. a. What is the purpose of the dft/A: data structure? 

b. How is data prepended to the beginning of a message? 

c. From what routines is a module's put procedure called? In whose context are service procedures called? 

3. a. When arc ^MCMe_r structures allocated? 

b. Explain why it is necessary for streams-oriented system calls to be serialized. What mechanism is used to 
perfonn this serialization? 

4. a. When a read or write system call is issued to an open stream, at what point does the flow of control first 

differ from the flow when an ordinary character-special device is called? 

b. Explain the role of the streams device driver in cloning. 

5. a. List the synchronization options available in streams. 

b. When is it permissible for a thread to block within a streams call? 

c. Explain how synchronization queues differ from ordinary streams queues. 

Advanced Question: 

6. Synchronization queues add another set of queues to a streams pipeline. Explain what effects this might have 
on data in the pipeline. For example, does die use of synchronization queues affect the order of messages? 
Does it affect the number of messages that may exist within a pipeline? 
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Module Contents 

1. Sockets 8-4 

Types of sockets 

Integrations into the operating system 

2. Mbufs 8-10 

Representing messages 

Memory allocation and liberation 

3. Implementation 8-26 

Protocol integration 

Socket data structure 
Parallelization 

4. Sockets and Streams 8-34 

Module Objectives 

In order to demonstrate an understanding of sockets and their use in supporting netwoiking in OSF/1, the student 
should be able to: 

• explain the difference between communication using datagram sockets and commimication using stream 
sockets 

• describe how messages are represented by mbufs 

• list the set of actions that may be taken when the list of free mbuf^ is exhausted 

• explain how kemel threads are used in the socket networidng subsystem 

• explain how OSF/1 supports both a streams and a socket interface to netwoiking 
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8-1. The Big Picture 



Networking in OSF/1 




EZD Mach 
EZ] UNIX 
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Student Notes: Networking in OSF/1 

OSF/l uses Berkeley's socket model to implement its communication protocols. The code is a parallelized 
version of that in 4.4BSD. The recommended user interface for networking is the X/Open Transport Interface 
(XTI), which is an enhancement of AT&T's Transport Layer Interface (TLI). While user code can communicate 
directly with the socket layer, XTI is more likely to be jjortable . 

OSF/1 implements XTI in a user-level library that communicates with the operating system via the streams 
interface. Since OSF/1 implements the network protocols in the socket framework rather than in the streams 
framewoik, the system provides a conversion layer. This conversion layer is a stream whose device driver 
converts stream requests into socket requests. 

Some of the material in this module is discussed in chapter 14 of Open Software Foundation, 1990a. 
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8-2. Sockets 



Interprocess Communication with Sockets 



• Sockets are an extension of the I/O interface for general-purpose 
interprocess communication 

• Sockets support a number of communication styles, as implemented by a 
variety of protocols 
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Student Notes: Interprocess Communication with Sockets 
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8-3. Sockets 



Socket Types 
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Student Notes: Socket lypes 

OSF/l includes only protocols supporting datagram and stream-type sockets. The UNIX domain contains both a 
datagram and a stream protocol, as does the Internet domain. (UDP is the datagram protocol; TCP is the streams 
protocol.) 
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8-4. Sockets 




Writing with Sockets 
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Student Notes: Writing with Sockets 

In Module 5 we saw that thefileops array contained the vector of entry points for operations on files. In this case 
it contains the vector of entry points for operations on sockets. 

The protosw contains the entry points into the selected protocol. 
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8-5. Mbufs 



Data Management 



8-5. 



• Streams 

— data passed by reference using mblks 

• Sockets 

— data passed by value and by reference using mbi^ 
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Student Notes: Data Management 

As with streams, data structures must be provided to facilitate the efficient movement of data through the various 
layers of the system. The data structures used with sockets, known as mbujs, are very similar to the data structures 
used with streams (mblks). The major difference is that while with mblks all data is passed by reference, with 
m^M/s small amounts of data can be passed by copying (though larger amounts of data are passed by reference). 
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8-6. Mbufs 



The mbuf Structure, part 1 
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Student Notes: The mbuf Structure, part 1 

Mbi^ either contain a small amount of data (ugtaiO&^^s) or refer to a laiger amount of data. As with mblks, a 
packet is represented by chaining a sequence of m^u/s, and a queue of packets is represented by chaining the first 
mbitf of each packet The header of each mbitf, besides containing the links in the various chains, describes the 
contents of the mbitfdnd includes a pointer to the first byte of data, wherever it may be. The first mbitfconxmns 
additional information: the length of the entire packet and possibly a reference to the netwoik interface fi-om 
which the packet came, or to which it is going. 
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8-7. Mbufs 



The mbuf Structure, part 2 
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Student Notes: The mbuf Structure, part 2 

If an mbitf does not contain its data but instead refers to it, then the m6u/"contains an m_ext structure to refer to the 
buffer. 



8-15 



Module 8 — Sockets 



8-8. Mbufs 



Virtual Copy (Socket Style) 
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Student Notes: Virtual Copy (Socket Style) 

Buffers may be passed b y referen ce; different layers of the protocol may each contain references to the same 
buffer. 
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8-9. Mbufs 



I'he Cluster Pool and Reference Counts 
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Student Notes: The Cluster Pool and Reference Counts 

The system maintains a pool of mbclusters, which are buffers to which mbi^ may refer. Thus, for example, to 
buffer data going to the network, the system allocates mbclusters to hold the data and mbujs to hold the protocol 
headers. 

Free mbclusters are linked together in a free list headed by mc^ree. The mclrefcnt array contains the reference 
counts to the mbclusters. A reference count of indicates no reference s to a particular mbcluster. A reference 
count of -1 indicates that no real memory is backing up the associated virtual address. 
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8-10. Mbufs 



Maintaining References 
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student Notes: Maintaining References 

Bu£fers need not be allocated from the mbcluster pool. In particular, the streams system can allocate buffers and 
pass them on to the socket system via the XTISO driver. The only difficulty in doing this is maintaining the 
reference count. The mclrefcnt array cannot be used, since these buffers arc not coming from the mbcluster pool. 

All mbi^ referring to the same buffer are doubly linked via their mjsxt structures. E^ch m_ext st ructure contains 
the address of a storage liberation routine to be called when the last reference to a buffer goes away. 
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8-11. Mbufs 



Mbufs from Mbclusters 
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Student Notes: Mbufs from Mbclusters 

The system allocates a fixed amount of virtual memory and a smaller amount of real memory for the pool of 
mbclusters. Free mbclusters (JosLCked up with real memory) are linked into a free list. When the supply of free 
mbclusters becomes too low, it is replenished with pages from the free page list. 

Mbi^ are obtained by allocating mbclusters and breaking them up into mbirfs. Free mbufs are linked together. 
When this supply of mbufs becomes too low, more mbi^ are allocated from mbclusters. 

The system invokes a garba2e collecto r every five seconds to examine the mbitflmbcluster situation and free up 
storage when necessary: if there are more than enough mbufs, then some arc coalesced back into mbclusters and 
returned to the free mbcluster list. If there are more than enough mbclusters, then some are deallocated by 
returning their pages to the free page list. 

The mclrefcnt array is used to aid the coalescing mbufs into mbclusters. When an mbcluster is on the free list, its 
reference count is 0. When it is broken up into mbujs, its reference count is set to 1. When an mbitf is allocated 
from the free list of mbufs, the reference count of the associated mbcluster is incremented by 1 (and decremented 
by 1 when the mbirf'is freed). 
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8-12. Mbufs 
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Student Notes: Responding to Memory Shortages 

Many protocols hold on to data maintained in mbi^ for a period of time. For example, the IP protocol maintains 
a reassembly queue to hold on to pieces of fragmented packets. When further pieces arrive the IP protocol 
reassembles them into a whole packet. If the operating system is extremely short of real memory, it calls each 
protocol's drcdn routine to ask the protocol to liberate as much memory as possib le. In response to a call to its 
drain routine, the IP protocol releases the mbufs in all of its reassembly queu gsr 
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8-13. Implementation 



IPC/Networking Control Flow 
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Student Notes: IPC/Networking Control Flow 

In a typical protocol stack, the top-level protocol might be TCP, the bottom-level protocol IP, and the network 
interface an ethernsunterfscc. A user thread making a system call enters the socket layer and calls the top-level 
protocol's usrreq entry point Control may continue via a call to the next level of protocol, which may then queue 
an output request by calling the network interface. 

Packets coming from the network arc dealt with in the interrupt context by the network-interface module, and then 
queued for processing by the bottom-level protocol. A kernel thread, one of a number of netisr threads, caUs the 
bottom-level protocol's intr entry point and then processes the queued packets. The kernel thread might then 
continue by calling the top-level protocol's input entry point. This protocol processes the data, which is queued 
on the socket either immediately or sometime later. 

The protocols will also be called periodically by netisr threads to handle timeouts. Because the netisr threads may 
block, there should be a few more of them than there arc processors (so as to maximize the utilization of the 
available processors). 
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S-14. Implementation 



Socket Data Structure 
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Student Notes: Socket Data Structure 

All of the fields of the socket data structure are illustrated in the picture. Of particular importance to this 
discussion are the socket send and receive queues (so_snd, so_rcv) of type struct sockbitf. This data structure 
contains: 

1. a byte count 

2. a high-water mark (the maximum allowable size of the queue) 

3. a message count 

4. a maximum message count 

5. a low-water mark (used for flow control: any thread blocked waiting for a space in the queue is woken when 
the queue size drops below the low- water maik) 

6. a pointer to the queue itself (a linked list of mbu^) 

7. the address of a routine to be called to wake up anyone blocked on the queue 
In addition, there is a blocking lock on the entire socket structure. 
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S-15. Implementation 



Protocol Control Blocks 
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Student Notes: Protocol Control Blocks 

This picture illustrates the protocol control blocks used by TCP/IP. Each connection is represented by a pair of 
protocol control blocks: an Internet PCB and a TCP PCB. The Internet PCBs of all the active TCP connections 
are doubly linked (necessitating a sequential search to associate a packet with a connection). 
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S-16. Implementation 



elizing TCP/IP 



Each domain may supply aLfimnel to specify processor constraints 



— the OSF/1 implementation of the Intemet protocols has no such 
constraints 

Blocking locks on socket and Internet PCB structures, simple locks on 
interface queues 
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Student Notes: Parallelizing TCP/IP 

Associated with each family of protocols (domain) is sl funnel structure, as discussed on page 2-103.. 

The Internet family of protocols (TCP/UDP/IP, etc.) have been parallelized. Blocking locks are in the socket and 
Internet PCB structures. The locking order is first to acquire the socket lock and then, when necessary, acquire the 
Internet PCB lock. Since these locks are blocking locks, they must be acquired by threads. 

Simple locks are used on data structures such as the interface queues, which must be accessed in the interrupt 
context. 



8-33 



Module 8 — Sockets 



8-17. Sockets and Streams 
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Student Notes: Sockets and Streams 

A perhaps unfortunate fact of life is that there are two competing netwoik interfaces: streams and sockets. There 
are probably more applications cuirently built on sockets than on streams, but this may be changing. A modem 
UNIX system should be able to support both interfaces. 

SVR4 does so by providing a socket emulation library (in user mode). OSF/1 . which supports both sockets and 
s geams in the kemel, has no need for an emulation library, but instead provides a means whereby a protocol 
implemented in the socket framework can be accessed via the streams interface. 
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8-18. Sockets and Streams 
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Student Notes: XTISO 

XnSO (X/QpenT Yansport Interface to Sockets) is a technique for supporting a streams-based XTI library with a 
socket-based protocol implementation. The XTISO stream consists of a standard stream-head module, a fairly 
simple timod module, and the XTISO driver module, which is the interface to the socket level. The XTISO driver 
takes transpoTjiinterface (TPI) messages and converts them into operations on sockets. This transfomiation 
requires converting messages represented as mblks into messages represented as rnbitf&. These messages are 
passed to the/socket layer by calling the socket code as if a socket system call had just been made. 

Messages coming up from the transport layer are queued on the socket's receive queue. Normally the next step is 
to wake up any process waiting for this message. What happens instead is that the XTISO driver's read service 
procedure is enabled. This procedure is called to pull messages represented as mbuj^ from the socket's receive 
queue, converts them into messages represented as mblks, and converts these into a TPI message which is sent 
upstream. 

The timod module converts messages sent to it from the XTI library via the stream head into the TPI format. This 
conversion primarily involves changing lOCTL-type messages into PROTO-type messages. Upstream messages 
are converted fiom PROTO-type into lOCACK-type as required. 
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Exercises: 

1 . What are the differences between communication tiirough datagram sockets and communication through 
streams sockets? 

2. a. Why do some mbirfs contain data while others point to data that is external to the mbup. 

b. How is it determined whether any mbufs refer to a buffer? 

c. List the set of actions that may be taken when the list of free mbufs is exhausted. 

3. a. How are kernel threads used in the socket and networking subsystem? 

b. Which socketAietworking data structures need locks? What types of locks are required? What is the 
precedence relation of these locks? 

4. How does the XTISO driver serve as an interface between streams and sockets? 
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Module Contents 

1. Role of the LVM 9-4 

Physical volume spanning 

Mirroring 

Logical volume resizing 

Bad-sector remapping 

2. Data Structures 9-12 

Logical volumes 

Physical volumes 

3. Components and Flow of Control 9-26 

Request serialization 

Consistency management 
Logical-to-physical mailing 
Request scheduling 
Handling bad sectors 

Module Objectives 

In order to demonstrate an understanding of the concepts and implementation of the logical volume manager, the 
student should be able to: 

• list the functionality provided by the LVM but not provided by the standard file systems and disk device 
drivers 

• explain how the LVM can be certain that it has an accurate description of a volume group, even when some of 
the underlying physical volumes are inaccessible 

• hst the benefits of mirroring 
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9-1. The Big Picture 
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Student Notes: Logical Volume Manager 

This material is discussed in chapter 15 of Open Software Foundation, 1990a. 
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9-2. Role of the LVM 



Logical Volume Manager (LVM) 



• 



A layer fitting between physical volumes and file systems 

— presents a device-driver interface to the file system 

Provides the notion of a logical volume: 

— logical volumes may span multiple pl^ ical volumes 

— logical volumes may bQ irtirrors iLon multiple physical volumes 

— logical volumes m^^^grow an dshriij c under the control of the^-^ 
administrator 

— logical volumes support software bad-sector remapping 
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Student Notes: Logical Volume Manager (LVM) 

UNIX files have always been limited by their inability to span multiple volumes. Since logical volumes can span 
multiple physical volumes, this restriction is pretty much removed in OSF/1. 
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9-3. Role of the LVM 



Logical Volume Manager Organization 
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Student Notes: Logical Volume Manager Organization 

Logical volumes are organized within volume groups that contain both logical volumes and physical volumes. 
Logical volumes arc divided into logical extents, the size of which may be any power of 2 between 1Mb and 
25 6Mb. Each logical extent is mapped to one, two, or three physical extents on physical volumes. The size of 
physical extents is equal to the size of logical extents, which is the same throughout a volume group. Logical 
volumes appear to be real devices to most of the system, so they have a name as a special file within the /dev 
directory. 

Each logical volume contains a single file system. The size of the logical volume may be easily changed by 
adding or removing logical extents and associating with them physical extents. In OSF/1 release l,neiflie.r the S5 
nor the UFS file system su pports the notion o fgrow^ or shrinkage in a file system's underlying volume. 
However, in OSF/1 relea se 1.1 these file systems will be ' ^rowable." 

If set up properly, the oiganization of logical volumes will not interfere with the organization of the UFS file 
system. Each UFS file system is built with the assumption that each cylinder group is composed of contiguous 
cylinders, but there is no built-in assumption that adjacent cylinder groups are acmally near one another. Thus the 
UFS disk-aUocation strategies will continue to woik as long as cylin dcrffloupsdo not cross logic^ _^tgDt 
boundaries. 
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9-4. Role of the LVM 
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Student Notes: Mirroring 

There arc two motivations for mirroring: speed and crash recovery. 

Mirroring can be used to speed read accesses to a logical volume: read requests to a mirrored logical extent can be 
translated into reads of a physical extent on the least busy physical volume. This approach is particularly useful 
for logical volumes that are "read-mostiy,** such as a logical volume containing binaries. 

Most importantiy, disk mirroring provides sufficient redundancy to survive crashes. If a physical volume is lost, 
the data contained in it can be recovered from copies maintained in other physical volumes — the mirrors. It is 
very important to insure that all minors containing the same data are identical. If an update was in progress at the 
time of a crash, only one mirror may have been updated. Recovery procedures arc needed to reestablish the 
consistency of the data. Since it may not be known which mirror is the most recent, the recovery procedures 
select one mirror and copy it to the others. Thus the primary goal of recovery is to regain consistency among the 
mirrors. 
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9-5. Role of the LVM 



Bad Sector Remapping 



9-5. 



• Augments remapping provided by the device and its driver 

• Hard errors 

— offending sector is remapped 

— sector copied from mirror if possible 

• Soft errors 

— offending sector is verified and remapped if necessary 
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Student Notes: Bad Sector Remapping 

The logical volume manager (LVM) augments the bad-sector remapping provided by the hardware. For mirrored 
volumes, the LVM can fix newly detected bad sectors by relocating the sector, reading the mirror, and writing the 
data into the relocated sector. 

If a "soft" read error occurs (an error that was detected and corrected by the disk controller), the data is rewritten 
and verified. If a "hard" error now occurs, then the offending sector is remapped. Errors are not passed back to 
the file system unless a hard error occurs on a read from an unmirrored physical extent. 
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9-6. Data Structures 



Logical Volume Manager: Physical Volumes 




- physical volume reserved area -^/^ 

— volume group reserved area-^-/^ 

— user data 
I — common bad-sector relocation pool 



physical volume 
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Student Notes: Logical Volume Manager: Physical Volumes 

A physical volume may be either an entire physical disk or a portion of a partitioned disk. That is, physical drives 
may be partitioned as they have always been in UNIX, or volxmie groups can effectively partition the physical 
volumes. By physical volume, from here on, we mean either a portion of a partitioned disk drive or the entire disk 
drive, depending upon how the drives are organized. 

A certain amount of overiiead is required within each physical volume. This consists of: 

• physical volume reserved area 

— this describes the individual physical volume 

• volume group reserved area (VGRA) 

— this describes the entire volume group 

• common bad-sector relocation pool 

As an option, space for bad sectors may be reserved at the end of each physical extent. This reduces the long 
seeks that would otherwise be required for remapping. 
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Student Notes: Physical Volume Reserved Area (PVRA) 

• At a fixed location on the volume (must be good sectors!) 

• Identifies the physical volume 

• Gives its size 

• Gives layout of rest of volume 

• Contains bad-sector directory 
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9-8. Data Structures 



Volume Group Reserved Area (VGRA) 
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student Notes: Volume Group Reserved Area (VGRA) 

• Volume group descriptor area (VGDA) 

— identifies all physical and logical volumes in volume group 

— gives mapping (for entire volume group) of physical extent to logical extent 

• Volume group status area (VGS A) 

— lists missing/present status of each physical volume 

— lists stale/ok status of each physical volume's physical extents 

• Mirror consistency record (MCR) 

— lists updates in progress 
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9-9. Data Structures 



Volume Group Descriptor Area (VGDA) 
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student Notes: Volume Group Descriptor Area (VGDA) 

Usually each physical volume contains two copies of the VGDA, one of which is up to date. However, if there are 
many physical volumes in the volume group, it would be excessively redundant for each physical volume to 
contain copies of the VGDA,josomp may haygjio^o^cs^fit. 

Each copy of the VGDA contains a timestamp. The copy with the most recent timestamp is considered to be the 
valid copy. When the VGDA is to be modified, the older copy of the two (per physical volume) is modified and 
thus becomes the newer version. To protect against failures while updating the VGDA, two timestamps are used, 
one at the beginning and one at the end of the area. The sectors containing the VGDA are written out 
synchronously and in order. When the VGDA is read, if the beginning timestamp and the ending timestamp are 
not equal, then a failure must have occurred during the update, and this copy of the VGDA is invalidated. 

To guarantee consistency with the volume group, a quorum (more than hal^ of physical volumes must have 
identical VGDAs. If such a quomm is not available (i.e. some physical volumes are "down**), then no operations 
are pemiitted that would result in updating the VGDAs. 
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9-10. Data Structures 
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Student Notes: Volume Group Status Area (VGSA) 

Each physical volume that has a VGDA has two copies of a volume group status area (VGSA) indicating whether 
the physical volume is available or missing. Like the VGDA, each of these copies is timestamped at the 
beginning and end. The VGSA is of particular importance after the system resumes operation following a crash, 
since the system must detemiine which of the volumes that had been available are now missing. In addition, ibis 
area indicates, for each physical extent within each volume, whether it is stale oro^. A physical extent on an 
available volume marked stale must be made ok by copying into it an ok version of its data. 
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9-11. Data Structures 



Mirror Consistency Record (MCR) 
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Student Notes: Mirror Consistency Record (MCR) 

The mirror consistency record (MCR), like the VGSA, is maintained to help restore consistency following a crash. 
It indicates which portions of the logical volume were being modified at the time of the crash. Each physical 
extent is divided mio logjog ltrack groups , as will be discussed. 
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9-12. Data Structures 
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Student Notes: Representing a Logical Volume in Primary Memory 

The extent maps are the inverse of those provided in the VGDAs, i.e., they provide mappings from the logical 
extent to the physical extent. 

The struct pvols contain status infomiation about each of the physical volimies and their physical extents. 
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9-13. Components and Flow of Control 
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Student Notes: Flow of Control 

The primary purpose of the strategy layer is to synchronize requests with respect to changes in progress at lower 
levels. For example, while a sector is being relocated, the strategy layer blocks all further requests for that sector. 
This sort of synchronization is accomplished by serializing requests at the strategy layer whenever a new request 
arrives, it is held up until all earlier requests to overiapping sectors have been completed. When administrative 
commands change information in the VGDA regarding a particular logical volume, all operations on that volume 
are held up until the change is complete. 

The mirror consistency manager maintains the consistency of mirrors. It keeps a list of update operations in 
progress, so that it can regain consistency in the event of a crash. 

The scheduler layer is responsible for translating logical requests into physical requests. If the logical voliune is 
mirrored, then each logical request may correspond to two or more physical requests. 

The physical layer is responsible for bad sector relocation. It communicates dircctiy with the real device driver 
and responds to disk errors by relocating sectors. 
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9-14. Components and Flow of Control 
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Student Notes: Consistency Management 

The basic approach to consistency management is to maintain a record on disk of the write operations currently in 
progress to mirrored extents. Thus, after a crash, those operations that were in progress can be identified and the 
minors involved can be made consistent with one another. A straightforward but amazingly expensive 
application of this approach would be to update this MCR before and after each disk write. The approach taken 
instead minimizes the number of extra disk writes at the expense of a longer crash recovery procedure. 

An up-to-date listing of updates in progress is kept incore. The on-disk record, however, lists update operations as 
being in progress for some time longer than they actually are in progress. 

The incore data structure for representing update operations in progress is the mirror -write consistency cache 
(MWC). This cache contains 62 entries. Each active entry is marked either "clean" or "dirty". The clean entries 
are arranged in LRU Geast recently used) order. Whenever an update operation starts, an entry representing a 
logical track group (LTG) in a logical event is allocated in the MWC and mariced "dirty". Each physical volume 
contains a recent copy of the MWC called the mirror consistency record. These copies are timestamped; the copy 
with the most recent timestamp is the valid one. Whenever a new entry is added to the MWC. the contents of the 
MWC are copied to the MCR of at least one of the physical volumes involved in the update. The MCRs contain 
no indication of "dirty" or "clean"; thus all entries marked "clean" in the MWC are interpreted as "dirty" in the 
MCR. 

Each physical extent is divided into logical track groups o f 32 pages each, where a "page" here is the disk block 
size. Each entry in the MWC and the MCRs corresponds to an LTG. Before any portion of an LTG is modified, 
an entry for it is allocated in the MWC and mariced "dirty," and a new MCR reflecting the updated MWC is 
written. When the update completes (to all mirrors), the MWC entry is marked "clean" but no new MCR is 
written (thus if the system were to crash at this moment, the MWC would disappear and the most recent mirror 
consistency record would indicate that an update to this LTG is still in progress). If the LTG is updated again, 
then the MWC entry is changed to "dirty" but, again, no new MCK needs to be writtea At some point there will 
be a period of no updates to the LTG, and its MWC entry will become the least recently used clean entry. When 
the next update request arrives for any other LTG, this entry is used to represent the new LTG and a new MCR 
will reflect this change, with the effect that the original LTG is no longer indicated as having an update in progress 
in the most recent MCR. 
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9-15. Components and Flow of Control 
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Student Notes: Consistency Management: Crash Recovery 

When a crash occurs, some number of update operations may be in progress. If a mirrored volume was in the 
process of being updated, the crash may cause the mirrors to be different. The mirror consistency manager must 
restore consistency, i.e. make mirrors identical with one another. 

Crash recovery begins with locating the most recent MCR, which contains a list of the LTGs that were being 
modified. Since it is not known which mirror of each LTG contains the most recent information, the mirror 
consistency manager chooses one arbitrarily (actually, it issues a read request to the logical volume: the scheduler 
layer, using its rules for read scheduling, reads the data from a single LTG) and copies this LTG onto each of the 
others. 

Another possible problem is that a physical volume may become unavailable (e.g., because of controller or media 
failures). This physical volume might contain only the most recent version of the MCR. If this volume was lost 
in a crash, then it cannot be known whether or not it contained die most recent MCR, so we must assume that it 
did. To cope with this, OSF/1 assiunes the worst case: that every LTG on the volume was being modified at the 
time of the crash. We then must indicate that all associated mirrors may be inconsistent This is done at the 
physical-extent level by maiking all physical extents of this volume as "stale" in the VGSAs. Then, for all logical 
extents that were double-mirrored, resynchronization is done by copying one of the accessible physical extents to 
the other (the second physical extent is also marked stale). If the logical extent was single-mirrored, then only one 
accessible physical extent remains, which by definition is consistent (Due to the drastic steps taken for recovery 
when a physical volume becomes unavailable (e.g., an entire disk might be copied), this procedure is performed 
only if explicitiy pemiitted by the operator.) 
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9-16. Components and Flow of Control 
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Student Notes: The Scheduler Layer 

The scheduler layer converts the logical request represented by aT^ttf structure into one or more physical requests 
represented by p^u/* structures. For a write request, all mirrors must be modified; in a read request it is only 
necessary to access a single mirror. Two general policies are used for scheduling the requests: sl parallel and a 
sequential policy. 

In the parallel policy, a read request is always sent to the physical volume with the fewest outstanding I/O 
operations. Write requests are issued in parallel to the physical volumes. 

In the sequential policy, a read request is performed by attempting reads from mirrors in a predefined order. If 
the first read succeeds, then the operation completes; otherwise a read from the next physical volume is 
attempted, and so on. Write requests are performed sequentially; a write request to one physical volume is 
complete before a write request to the next physical volume begins. 

Clearly, the parallel policy is more efficient than the sequential policy. However, the sequential policy is safer. 
For example, if the system crashes in the middle of an update, the updates of each of the mirrors are affected with 
the parallel policy, while with the sequential policy, only the update of one of the mirrors is affected. 
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9-17. Components and Flow of Control 



The Physical Layer 
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Student Notes: The Physical Layer 

The physical layer*s main responsibility is to handle bad-sector remapping. It must examine each physical request 
for references to any known bad sectors. If a request does refer to a bad sector, then (assuming only a single bad 
sector for ease of exposition) the physical layer breaks the request into three pieces: the first piece for the request 
up to the bad sector, the second piece for the relocated bad sector, and the third piece for the remaining portion of 
the request. The pieces are then treated as separate requests and are processed sequentially. 

The device driver detects new bad sectors during an I/O operation and reflects the error back to the physical layer. 
There are essentially two types of errors: soft errors and hard errors. Soft errors have been detected and corrected 
by the disk controller. The physical layer attempts to test the sector by sending a "write-verify" request to the disk 
driver. If this succeeds, then nothing else need be done. However, if it fails, the sector is remapped. 

If a non-mirrored read operation encounters a hard error, then there is no choice but to reflect the error to the 
caller. However, an entry is made in the bad-block directory indicating that relocation is desired and this is done 
the next time this sector is written. 

If a mirrored read operation encounters a hard error, the scheduler layer perfomis a read to a mirror, then sends a 
write request to the original sector specifying that hardware relocation is desired. The physical layer passes this 
on to the device driver. If hardware relocation fails or is not supported, then software relocation is performed. 
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Exercises: 



1 . a. What functionality does the LVM provide that the standard file systems and disk device drivers do 

not? 

b. Which LVM-related information is replicated over most of the physical volumes? Which is only 
maintained on a single physical volume? 

2. a. What is the purpose of timestamps in the on-disk data stmctures? 

b. How can the LVM be ceitain that the volume group descriptor information is valid even when some 
physical volumes are inaccessible? 

3. a. Why is it necessary to serialize overiapping requests? 

b. What information is contained in the MCR? 

c. Suppose that, as part of crash recovery, it is discovered that the MCR contains a single entry and this 
entry pertains to a doubly mirrored volume. Assuming that all physical volumes are present, list the 
actions taken as part of the recovery of the logical volume group. 

d. Explain the differences between the parallel and sequential scheduling policies. 
Advanced Question: 

4. What must be changed in the UFS file system so that it can exploit the ability of logical volumes to grow? 
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Module Contents 

1. Role of the Loader 10-4 

Loader functions 

Interaction with exec 
Shared libraries 

2. Symbol Resolution 10-10 

Packages and libraries 

Known package tables 

3. Data Structures and Flow of Control 10-18 

Format-independent aspects and format-dependent aspects 

4. The Run-time Image 10-22 

Relocation and shared libraries 

Address space layout 

5. Dynamic Loading 10-26 

Loading into the current process 

Loading into the kernel 

Module Objectives 

In order to demonstrate an understanding of shared libraries, dynamic loading, and symbol resolution, the student 
should be able to: 

• describe the functionality provided by the OSF/1 loader that is not provided by traditional Id and exec 

• explain the role of packages in symbol resolution 

• explain the phases of the run-time load procedure 

• list the three techniques for implementing shared libraries and their advantages and disadvantages 

• describe the differences between loading into the current process and loading into the kernel 
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10-1. The Big Picture 



The OSF/1 Loader 
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Student Notes: The OSF/1 Loader 

The material in this module is discussed in chapter 6 of Open Software Foundation, 1990a. Additional 
infomiation about the OSF/1 loader can be foimd in Allen, 1991. 
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10-2. Role of the Loader 



Loader Goals 
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Student Notes: Loader Goals 

A library is a collection of modules contained in a file. A module is a component of an executable image. For 
example, a module might be a result of compiling or assembling a file containing a program in source form. 
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10-3. Role of the Loader 



The exec System Call and the Loader 



• Standard exec 

• Riin-time loader 



10-3- © 1990. 1991 Open Software BDondttian 



10-6 



Module 10 — Loader 



student Notes: The exec System Call and the Loader 

As in any UNIX system, the exec system call can invoke a program. In older UNIX systems, such a program 
must have been ful ly bound an d relocate d. While OSF/1 certainly supports this mode of operation, the system 
also allows much of the binding and relocation to be postponed until run time. 

When a program is exec'd, the system-call handler examines the program: if the program is in a recognizable 
load format, is fully relocated and contains no unresolved references, then it is loaded directly and control is 
passed to it on return to user mode (as usual). Otherwise, exec loads the (user-mode) ngi^ ^me load erjDtgtiie 
a ddress spac e an d passes control to it on re t urn to user mode;_giving jLthe name .of th e program to jje loaded. The 
run-time loader then completes the load process. It copes with load formats unrecognized by the kernel, linking 
the image to shared libraries, loading additional modules as required, and relocating the entire image as necessary. 
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10-4. Role of the Loader 



Constructing an Executable Image 

• Linking A<^ 

— matching symbols to packages 

• Loading /^^Kgc 

— symbol resolution 

♦ pull in all necessary modules 

— relocation 
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Student Notes: Constructing an Executable Image 

The first step in the creation of an executable image is the construction of the component modules. These 
modules are created by linking together the object code produced by compilers and assemblers. Some of the 
symbols referenced in the resulting module might not be defined there, but are instead defined in some library. In 
older UNIX systems, the routines from the libraries that define these symbols would be copied and bound into the 
resulting module. With the OSF/1 loader, the linker may merely augment the symbol name with the library name 
(package name) that defines it, and postpone until later fetching the routine deHning the symbol. Thus the result 
of linking is to create an imported symbol table that contains a list of unresolved symbol-name/package-name 
pairs. 

The loading procedure may be completed at run time. The first step involves final symbol resolution. AU of the 
packages mentioned in the imported symbol table must be tracked down and the routines containing the 
unresolved symbols must be extracted. This is in general an iterative procedure, since the routines so extracted 
may have their own imported symbol tables, causing further symbol resolution. As symbol resolution is 
performed, relocation is also performed. 
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10-5. Symbol Resolution 



Packages and Libraries 



• Two-dimensional name space 

— package name/symbol name 

• Packages are mapped to libraries at nm time 

— binaries need not contain path names 
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Student Notes: Packages and Libraries 

The concept of packages was invented to deal with the following issues: 

• symbols should not be bound to library path names, since libraries may move 

• naming conflicts may occur between symbols of different libraries 

• symbol resolution should Inflexible: the user should be able to alter resolution at compile, link, and load time 

Packages are the abstraction of libraries: they normally correspond to libraries, but the programmer is free to split 
a library into sub-libraries by breaking it up into a number of packages. Currently, package names are assigned to 
symbols only at link time, but with sufficient compiler support they could also be assigned at compile time. 
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10-6. Symbol Resolution 



Symbol Resolution 
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name: printf 
pkg: nml 
value: null 




name: printf 
pkg: stdio 
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name: printf 
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Student Notes: Symbol Resolution 

Prog.o is produced by the cx)mpiler. Its imported symbol table contains the names of unresolved symbols, but 
little is known about these symbols. At link time, the Id program finds definitions for the symbols in libraries, but 
instead of extracting the code, it augments the imported symbol table by filling in the package for each of the 
symbols listed. Finally, at load time, the Id program somehow brings the desired packages into the address space 
and fills in the value fields of the imported symbol tables with the addresses of the component routines. 
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10-7. Symbol Resolution 



Known Package Tables (KPTs) 
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Student Notes: Known Package Tables (KPTs) 

Known package tables (KPTs) contain mappings fix)m package names to actual code. They arc used to translate a 
package-name/symbol-name pair into a symbol within a particular library. Each process has a sequence of KPTs 
which it searches to resolve a particular package-name/symbol-name pair. The order of search is: 

1. loaded package table (LPT): a per-process table referring to packages from modules that have been explicitly 
loaded into the program; this includes the loader itself, the "main" module (whose name was given in exec), 
modules loaded because they arc dependencies of other modules, and modules that have been dynamically 
loaded 

2. private known package table (private KPT): maintained by the user and inherited copy-on-write from the 
parent process (this is contained in anonymous memory that is retained across execs) 

3. global known package table (global KPT): a system-wide table maintained by the system administrator (used 
to define the standard system libraries) 
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10-8. Symbol Resolution 



Package Substitution 

prog.o prog libca 
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Student Notes: Package Substitution 

By modifying the contents of the private KPT, the user can effect changes in symbol resolution. In this example, 
by inlitnng mylib.a, we have created a private package containing a redefinition of printf. Since at link time the 
symbol printf in. our program has been associated with the package stdio, we have to call our private package stdio 
as well, so that it is used to define the pr/w//* mentioned in our program. Our private stdio package, containing 
only printf, will appear in the private package table. 

At load time, since the program searches the private KPT before the global KPT, it uses the new version of printf. 
However, any reference to any other member of the standard stdio package is satisfied via the global KPT, since 
we have only replaced pn>i(/' of the standard stdio package. 
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10-9. Data Structures and Flow of Control 



The Run-Time Loader 
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Student Notes: The Run-Time Loader 

This picture gives a simplified view of the data structures used by the loader. There is a context structure for each 
image maintained by the loader. Typically there will be just one context structure, representing the image in the 
current address space. However, the kernel loader, for example, would have a context structure for another 
image — ^the kernel image. The context structure is initialized to refer to the global KPT and the inherited private 
KPT. The LPT is initialized to refer to the loader itself. Each module loaded into the address space is represented 
by a module record structure, which is linked into the known module list, which is headed by the context structure. 
Each module record points to an array listing all of the packages contained in the module and made available to 
other modules (exported). Associated with the private and global KPTs are module records for each of the 
modules supplying packages for the associated KPT. Unlike the module records in the known module list, which 
contain detailed information about the module such as the imported symbol table, these module records only list 
the exported packages of the module. 

Initially the known module list contains a module record for the loader. The next step is to append a module 
record for the main routine, i.e., the one given in the exec system call. The goal of the loader is now to build a 
complete known module list, containing all modules that are required for the image. It does this by identifying for 
each module record in the list the additional modules it needs and adding the record for these additional modules 
to the known module list. 

The packages exported by these modules are added to the loaded package table. Modules are added to the known 
module list when they are needed to resolve symbols listed in a module's imported symbol table. Module records 
for these modules are appended to the end of the known module list. In some load formats, the names of these 
modules are supplied explicitly. The technique intended is that the symbol-name/^ackage-name pairs given in the 
unported symbol table will be looked up in the loaded package tables, as discussed on page 10-15. The next step 
is for the loader to traverse the known module list and to map in (if not already mapped) the regions (e.g., text, 
data, BSS) of each module. At the same time the loader can determine the values of each module's exported 
symbols. 

Finally, the loader traverses the known module list again and performs relocation in each module. 

In summary, run-time loading consists of three phases: 

/^^ discovery — ^locating the desired modules based upon translating symbol-name/package-name pairs to routines 
in libraries 



(j) • mapping — ^map the modules into memory 
" //) • relocation— conyeit symbolic reference to actual addresses 



10-19 



Module 10 — Loader 



10-10. Data Structures and Flow of Control 



Multiple Load Formats 



• Fonnat-dependent loader 

— format recognition 

— construction of imported symbol table 

— provision of exported symbol table 

— mapping of regions 

— relocation 

— imloading 
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Student Notes: Multiple Load Formats 

Associated with each object forniat is a set of routines known as the format-dependent loader. Adding a new 
object fomiat merely involves writing another version of these routines. The primary duties of these routines 
include: 

1. format recognition: whenever the loader encounters a new module, it calls upon each of the fonmat-dependent 
loaders in turn, essentially asking them '^ this one ofv otKS?" 

2. construction of imported symbol table: each module's format-dependent loader is called upon to fill its 
imported S3anbol table 

3. provision of exported symbol table: each module provides a list of exported symbols in format-dependent 
form. Thus each module also provides a routine to return the value of each of the symbols it defines 

4. mapping of regions: each module's format-dependent loader maps the regions of the module into memory as 
required 

5. relocation: the format-dependent loader performs all necessary relocation 

6. unloading: if the module is to be unloaded, die format-dependent loader performs the necessary chores 

A detailed discussion on the format-dependent portion of the loader can be found in chapter 7 of Open Software 
Foundation, 1990b. 
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10-11. The Run-time Image 



Shared Libraries 



process A 



printfi...)' 



stdio 



process B 



printfi...) 
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Student Notes: Shared Libraries 

Shared libraries is one of the most important features of the OSF/1 loader. There are a variety of ways to share 
code. The simplest is merely to map the shared routine into the address space. If relocation is required, the 
routine may be mapped copy-on-write and then appropriately modified to effect the relocation. However, if 
appreciable relocation is required, this technique cuts down on the amount of sharing that actually takes place. 

Another approach is to use position-i ndependent code (PIQ to eliminate the need for pre-relocation. This 
technique is not currently supported by the OSF/1 compilers, but might be supplied by vendor-supplied compilers. 

Wha t OSF/1 does is to use iMg-rg1nr.atp.(1 Rhargdjjhrarifts Such libraries are combined into a single image which 
is pre-rclocated to fit at a fixed location in the address space. They are thus available for linking to programs 
without any further relocation. The standard libraries are provided in this format and are linked in this form (as 
described on page 10-9) with the standard UNIX commands. 

Note that symbol substitution is still possible, e.g., an altemative version of printf can be substituted at load time 
for the version in the shared library. Also, because final symbol resolution can occur at load time, the locations to 
which shared routines have been prc-relocated can be changed without the need for relinking. 
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10-12. The Run-time Image 



Typical Address Space Layout 
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Student Notes: lypical Address Space Layout 
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10-13. Dynamic Loading 



Run-Time Loading 
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Student Notes: Run-Time Loading and Unloading 

Modules may be explicitiy loaded into or unloaded from a running program. For example, a user of a CAD/CAM 
application might put a transistor into a diagram. The CAD/CAM application might then call upon the loader to 
load a transistor-emulation module into the program. If the user subsequentiy decides to remove the transistor 
from the diagram, then the CAD/CAM application would call the loader to unload the emulation module as well. 

To make this possible, the run-time loader remains in the address space even after the program starts up. The user 
program can^c^l^gJoader via itsjoa^ and wntood entry points; all of the loader's data structures still exist, and 
they are updated by the loader to reflect the presence of the new module. 
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10-14. Dynamic Loading 



Kernel Loading 



• Handled b v kernel loader server 
— maintains loader data structures for the kernel 



— maps modules into its own address space but relocates with respect to 
kemers address space 

— copies relocated module into kernel via special system call 
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Student Notes: Kernel Loading 

The OSF/1 loader can be used to load modules into or unload modules from the kernel. This feature is used in 
conjunction with dynamic configuration to support loadableAuiloadable device drivers, streams modules and 
drivers, file systems, and protocols. Kernel loading is managed by a privileged user-mode task, the kernel-loader 
server. This server maintains the data stnictu res describing the kernel address space (i.e., the same types of data 
structures that describe a user task). 

Loading into the kernel is essentially identical to loading into a user task, except that the actual loading is done 
remotely: modules to be loaded into the kernel are mapped into the server's address space, but are relocated with 
respect to their final position in the kernel address space. Special system calls perform the actual loading into the 
system address space (such modules go into wired memory) and call the module's configure routine (so that it can 
link itself into kernel tables). 
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Exercises: 

1. What functionality is provided by the OSF/1 loader that is not provided by the standard Id and exec? 

2. a. Explain the role of packages in symbol resolution. 

b. How can one replace a routine supplied by a system library? 

3. a. Why can't the LPT and the private KPT be combined into a single table? 

b. Why is it necessary to separate the format-dependent loaders from the format-independent loader? 

4. a. List three techniques for implementing shared libraries. 

b. Why are loader text, data, and BSS kept separate from the standard text, data, and BSS? 

5. a. List the actions taken to load a module into a running program. 

b. In what ways is loading into the kernel treated differently from loading into the current process? 
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Module Contents 

1. Security Concerns 11-4 

Orange-book model 

Security in OSF/1 

2. Auditing 11-12 

3. Access Control 11-16 

Discretionary access control 

Mandatory access control 
Implementation architecture 

4. Authorizations and Privileges 11-28 

Representing authority 

Principle of least privilege 
Using and transferring privileges 

5. Living with Security 11-44 

Module Objectives 

In order to understand security in the OSF/1 environment, the student should be able to: 

• describe how OSF/1 is compliant with the Orange Book security model 

• describe how audit information is collected 

• explain how it is determined whether a particular subject has the desired access to a particular object 

• explain how authorizations and privileges exceed traditional security measures 
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11-1. The Big Picture 



Security 
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Student Notes: Security 

This material is discussed in chapter 16 of Open Software Foundation, 1990a. 
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11-2. Security Concerns 



Security Concerns 



PTOXecxfobjects)iiom subjects 

— subjects: users and processes 

— objects: files and processes 
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Student Notes: Security Concerns 
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11-3. Security Concerns 



The "Orange Book" Model 



• division D: minimal protection ^a() 



• division C: discretionary protection /LfKf 

— class CI: discretionary secunty protection ^{^ , /^ //'/<^ '^^^ /, / 
O^^^ — class C2: controlled access protection— ^,v^ytv /^^^^^^ 



divisions: mandatory protection -/^r^- 

— class Bl: labeled security protection 

— class B2: stmctured protection 

— class B3: security domains 

• division A: verified protection ^^t^^^K / ^ 

— class Al: verified design 
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Student Notes: The "Orange Book" Model 

The U. S. Department of Defense (DoD) trusted-computer-evaluation criteria, known as the "Orange Book" 
model (so called because of the color of its cover), define criteria for classifying computer systems according to 
their degree of protection. 

Division D contains those systems whose security features have been evaluated and have flunked. 

Division C, a very minimal level of security, contains two classes. To be in class CI , a system must provide 
controls so that users can protect private information and keep others from accidentally reading or destrojdng 
data. The model is of cooperating users processing data at the same levels of security. Most UNIX systems 
should fit within this class easily. 

To be in class C2, a system must meet all of the requirements for class CI and, in addition, users of the system 
must be individually accountable for their actions through login procedures, auditing, and resource isolation. 
UNIX systems may relatively easily aspire to be in this class. 

If a system is certified to be in division B, then it must be realistically considered secure. To be in class B 1, a 
system must meet aU the requirements for class C2. In addition, it must have an infonnal statement of the sec mjty 
model and must provide d ata label ing and mandatory access control over name d subjec ts ando^ects. The 
capability must exist fora ccurately labeling exported information (e.g., in a defense environment. Top Secret 
printouts must be clearly labeled as such). 

To be in class B2, a system must meet all the requirements for class B 1 but, instead of an informal statement of 
the security policy model, there must be a clearly defined and documented /oww/ security policy model. The 
discretionary and mandatory access controls of Bl must extend to all subjects and objects, much more thorough 
testing and review is required, and very stringent configuration management controls are necessary. 

To be in class B3, a system must meet all requirements for class B2. In addition, its trusted computing base 
(TCB), i.e. that portion of the system that runs in privileged mode, must be small enough to be subjected to 
rigorous analysis and test. AU accesses of subjects to objects must be mediated, the system must be tamper-proof, 
a jecurity admimsiEator must be supported, audit mechanisms must be expanded to signal security-relevant 
events, and detailed system recovery procedures must be in place. 

The primary difference between class Al and and class B3 is that the formally specified design must be foraially 
verified. Going beyond class Al, if such classes were defined, might involve a formally verified implementation, 
which is considered beyond the state of the art. 



11-7 



Module 11 — Security 



11-4, Security Concerns 



Compliant vs. Certified 



. ^^ 



• Certiflcatioii requires a Gengthy) fonnal evaluation process 
— platforms, not'l)perating systems, are certified 
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Student Notes: Compliant vs. Certified 

Certification requires a formal evaluation process. It is not merely the operating system that is being evaluated, 
but also the implementation of the operating system on a particular architecture with a given set of options. OSF 
supplies most of the voluminous documentation required for certification. 

The "Orange Book" criteria are strictly for stand-alone systems. A system that is networked cannot be secure 
under these criteria. Thus, for example, a Bl -certified syste m cannot contain NFS. 
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11-5. Security Concerns 



Security in OSF/1 



OSF/l can be compiled to be C2-compliant, B 1 -compliant, or neither 

Components: 

— aiditiag^<^2 

— discretionaiy access control (DAC) 0j 

— mandatory access control (MAQ fl I 

— authorizations and privileges / / 
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Student Notes: Security in OSF/1 
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11-6. Auditing 



Auditing 



• system calls 

— I/O 

— exec 

— fork 

— etc. 

• user events 

— login 

— su 

— etc. 
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Student Notes: Auditing 

Auditing may be used selectively; i.e., under the control of the system administrator, selected system caUs and 
user events may be recorded. 

Stub routines inserted in the control path collect infomiation for system calls. The status of each system call is 
maintained in the auditjrtfo stnicture, which is allocated on the kemel stack. When an audited system call 
completes, the auditjnfo structure is put into a buffer maintained by the audit device driver. This driver makes 
the infonnation available to the audit daemon, which compacts it and stores it in a database. 
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11-7. Auditing 



Kernel/Daemon Communication 




user 
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H module ^M 



kernel 
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Student Notes: Kernel/Daemon Communication 

A fair amount of the processing performed by the security system is done within user-level daemons. The 
security architecture was designed for general UNIX systems: it does not assume the communication facilities of 
OSF/1 . Thus communication between the daemons and the kernel must be implemented in a way that can be 
easily ported to any UNIX system. The interface chosen is that of apseudo device. The security daemons may 
perfonn standard read, write, and iocti system calls on their associated pseudo devices. These devices are 
represented in the kernel as pseudo-device drivers. 

Upcalls, e.g. requests sent to the daemons by the kernel, are implemented by having the daemon make a read 
system call. The call blocks until the pseudo-device driver has an upcaU to make. When the read returns in the 
daemon, the result contains the upcall request. 
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11-8. Access Control 



Discretionary Access Control (DAC) 



:, process ^ ^ 
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Student Notes: Discretionary Access Control (DAC) 

By discretionary access control we mean the ability of an object's creator/owner to specify who has what sort of 
rights to the object. In OSF/1 , the traditional UNIX security policy (a discretionary policy) has been augmented 
with the use of access control lists (ACLs).^ Associated witii each object (i.e. file) is a list of aU users allowed to 
access it and what their access rights arc. ACLs extend the nomial UNIX discretionary policy by increasing its 
flexibility. For example, with AQ-s it is easy to prohibit access to certain individuals. 
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11-9. Access Control 



Mandatory Access Control (MAC ) 
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Student Notes: Mandatory Access Control (MAC) 

Mandatory access control involves enforced restrictions on objects that cannot be changed at the discretion of the 
creator or owner or user. Subjects are assigned levels of trust (clearances) and objects are assigned degrees of 
sensitivity. Both notions are represented with sensitivity labels and combine a hierarchical classification with a 
non-hierarchical set of categories (or compartments). 

DoD security classifications are an example of a hierarchical classification — ^Unclassified < Confidential < Secret 
< Top Secret < Eyes-Only. Combined with this is the non-hierarchical notion of compartments, e.g. NATO, 
NORAD. Thus, to view a Top Secret NATO document, it is not enough to be cleared for Top Secret; one must 
also be woridng within NATO. 

Various rules are established governing access to information. For example, one cannot modify a Secret file while 
executing in a Top Secret domain, but one can read a Secret document while in a Top Secret domain. 
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11-10. Access Control 



Mandatory Access Control: Multilevel Directories 
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Student Notes: Mandatory Access Control: Multilevel Directories 

Public directories (such as Amp) need special treatment when used in conjunction with the mandatory access 
control policy. For example, various programs such as the C compiler use the Amp directory to hold their 
temporary files. Even though the contents of these files may be securely protected, the existence of a temporary 
file as well as its name could be easily discovered by anyone performing an Is on the /tmp directory. Such 
information about Top Secret temporaries, for example, should not be known to those operating in the Secret 
domain. 

Furthermore, if a direcdtory is, for example. Top Secret, then a process whose sensitivity level is Secret wouldn't 
be able to add files to it, since this would require modifications to the directory. If the directory was Secret, then a 
Top Secret process couldn't write to it, because this would allow the leakage of infonnation to a lower sensitivity 
level. 

One approach to dealing with this problem might be to change all applications so that they do not use public 
directories but instead find directories at appropriate security levels. A better approach is to deal with the problem 
transparently. A directory such as /tmp may be set up as a multilevel directory. It is then transparently split into a 
number of subdirectories, one for each security classification. Thus a reference to the file Amp/xyz by a process 
executing as Top Secret would be translated into a reference to the file AmpAopsecret/xyz. 
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11-11. Access Control 



Attributes 



• Examples: 

— UID/GID 

— security classification 

— security compartment 

— access control list 

• Each is represented by a 32-bit tag 
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Student Notes: Attributes 

Each security policy must deal with a set of attributes on subjects and objects. These attributes can be very 
complicated; for example, an ACX can be arbitrarily long. A direct representation of such attributes would be too 
complex for the kernel to manipulate easily. Instead, each attribute is represented by a 32-bit tag (thus when a 
new ACL is used, a new 32-bit tag is created). 

Every security policy has a policy daemon that is responsible for translating tags to attributes and vice versa. 
Each such daemon maintains a database to aid this process; it must ensure that tags are unique. Inside the kernel, 
the policy modules deal only with tags. Since they are not capable of interpreting the tags, they can only make 
equality comparisons; any other manipulation must be forwarded to the user-level policy daemon. 
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11-12. Access Control 



Tag Pools 



• Each security policy uses a fixed set of attribute types 

— DAC uses one each for subjects and objects (UID/GID, ACL) 

— MAC uses two for subjects (sensitivity level, clearance level) and one 
for objects (sensitivity level) 

• Each subject's and object's tags are organized into a tag pool 
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Student Notes: Tag Pools 

Each subject and object must be associated with its attributes with respect to each security policy. In OSF/1 , each 
subjea must have three tags (one for DAC and two for MAC) and each object must have two tags (one each for 
DAC and MAC). The collection of tags associated with a subject or an object is known as a tag pool. 



11-25 



Module 11 — Security 



11-13. Access Control 



Security Policy Architecture 




user mode 




security _switch 



kernel mode 
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Student Notes: Security Policy Architecture 

Whenever a security decision has to be made or action taken, each security policy must be consulted. In OSF/1, 
there are two such policies, DAC and MAC. Thus, for example, if a process attempts to open a file, both DAC 
and MAC are consulted to ensure that both will allow the access. If either says no, the access is denied. 

The security decisions arc implemented through the security switch. Each security policy provides 2l policy 
module in the kernel whose entry points are contained in the security switch. When a subject attempts to access an 
object, the macro SP_ACCESS is caUed, which calls the access entry point of each policy (via the security 
switch), passing to the policy module the subject's attributes and the object's attributes. Each policy module 
maintains a cache of recent security decisions, which consists of relations on attributes as represented by tags. If 
the decision cannot be made based upon the contents of the cache, then the policy module forwards the request via 
an upcall to its policy daemon, which then makes the decision. 
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11-14. Authorizations and Privileges 



Authorizations and Privileges 



11-14. 



• Authorizations 

— command authorizations are used to restrict certain subsystems and 
commands to designated users 

— kernel authorizations grant certain security policy overrides to trusted 
applications 

Privileges 

— rights to access operating-system functions 

— used partly to implement kernel authorizations 



• 
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Student Notes: Authorizations and Privileges 

A manage of an object finds out if the subject process is authorized by checking a list of authorized subjects, 
whereas a privilege is an actual property of the process, stored with the process. 
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11-15. Authorizations and Privileges 



Authorizations 



• Command authorization 

— specific 

— role 

— subsystem 
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Student Notes: Authorizations 

Command authorizations are associated with particular users and are maintained by user-level code and databases. 
These authorizations are broken into the following categories: 

• specific 

— allows the user to execute a conmiand to perform a specific functions. E.g., the mknod authorization allows 
the user to invoke the m^gg^ command to create special files 

— allows the user to perfoim tasks associated with some specific system role. E.g., the i§so authorization 
allows the user to administer the security system 

• subsystem 

— grants the user additional rights in certain subsystems. E.g., the Ip authorization allows the user to use the 
administrative and command options of the Ip subsystem 
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11-16. Authorizations and Privileges 



Privileges 



Root replacements 



• UNIX mode 



• Trusted mode 



• Trusted function 
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Student Notes: Privileges 

Instead of the supemser/mere-mortal dichotomy of UNIX, OSF/1 uses a much finer breakdown of privileges, 
dividing them into the following categories: 

• root replacements 

— a breakdown of the privileges once reserved for the root into a set of rights that can be individually granted; 
e.g., the sysattr privilege allows one to invoke system calls that change system attributes such as the time of 
day. 

• UNIX mode 

— privileges that ordinary users have in UNIX but may be restricted in OSF/1 ; e.g., one must have the exec ^uid 
privilege to execute SETUID programs. 

• tnistedmode 

— privileges allowing a process to operate in modes that gain it special treatment with respect to trasted system 
features; e.g., the suspendaudit pnwilQge allows a process to stop the kernel from collecting audit records. 

• tnisted function 

— privileges allowing a process to define new trusted functions; e.g., the writeaudit privilege allows a process 
to append records to the audit trail. 
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11-17. Authorizations and Privileges 



Use of Privileges 








Associated with the process: 
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• base privilege set 








• kernel authorizations set 








• effective privilege set 








Associated with the executable file: 






• potential set 








• granted set 
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Student Notes: Use of Privileges 

The base privilege set is the set of privileges that is always granted to a process when it execs a file. 

The kernel authorization set is the set of privileges for which the process's user is authorized. 

The effective privilege set is the set of privileges that are currently being used when the kernel checks for 
privileges. 

The potential set is the set of privileges that a program may use. 

The granted set is the set of privileges that is placed in a process's effective privilege set when the file is exeCd. 
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11-18. Authorizations and Privileges 



Exec'ing a File 
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Student Notes: Exec'ing a File 

Privilege sets are represented as bit vectors. Associated with each process arc four such bit vectors, for the base 
privilege set, the kernel authorization set, the effective privilege set, and ih& potential set. 

Each inode contains bit vectors for the granted and potential sets. 

When a user execs a program provided by another user, we have two mutually suspicious parties. Each party, 
i.e., the owner of the process and the owner of the executable file, provides an initial set of privileges that form the 
initial effective set of this joint venture. This effective set may be enlaiged, but only subject to constraints 
provided by both parties. Privileges can be added to the effective set that are either in the potential set or the base 
set. The base set can itself be enlarged, but only be adding to it privileges that are in both the kernel authorization 
set and the potential set. This protects both parties in the event that another file is exec'd. The base set used with 
this new file would contain only those privileges allowed by both parties. Tliis technique prevents privileges from 
being combined in unforeseen ways. 
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11-19. Authorizations and Privileges 



Privilege Set Relationships 



K: kernel authorizations 
B: base privilege set 
E: effective privilege set 




G: granted set 
P: potential set 
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Student Notes: Privilege Set Relationships 

A set of kernel authorizations defines the limit of what privileges this process is allowed to have. It is a subset of 
the privileges available to the user. The potential set limits the privileges of a process executing the program 
contained in a file. 

Be K 

Eg (PuB) 
GcP 
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11-20. Authorizations and Privileges 



Principle of Least Privilege /j2_ 



Trusted applications use the smallest effective privilege sets possible 



^^'^ ©1990, 1991 Open SaftwanRmaktiaQ 
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Student Notes: Principle of Least Privilege 

Good practice dictates that the effective privilege set should be kept as small as possible. The setpriv system caU 
allows a process to adjust its sets K, B, and E, subject to the constraints: 

K'c K 

B'e (KnP)uB 

E'g PuB 

(where ^, B\ and Ef are the sets K, B, and E after a setpriv request). 
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11-21. Authorizations and Privileges 



Operations on File Privilege Sets 



Processes that have the chpriv privilege may change a file's granted and potential sets 



11-21. 



* 1990. 1991 Open Software Rnmdatiaa 



11-42 



Module 11 — Security 



Student Notes: Operations on File Privilege Sets 

Using the chprv system call, a process may propagate to files only those privileges for which the process is 
authorized: 

P'c K 

G'c PnK 

If an executable file is modified, then all privileges are removed from its granted and potential sets. This is 
analogous to the effects of modifying a setuid file in standard UNIX. 
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11-22. Living with Security 



Living with Security 



• Secure systems are inherently .slQwet than non-secure systems 
— OSF/1 can be configured to be either C2 or Bl 
' — the degree of auditing can be set by the system administrator 
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Student Notes: Living With Security 

Not all installations will desire the Bl security features in OSF/1, The system can be configured to be either C2 or 
B 1 . This decision is implemented as a compile-time option; it is not a run-time option because too many tests 
would be necessary. However, the degree of auditing is selectable at mn time. 
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Exercises: 

1 . a. To which security class are most UNIX systems probably compliant? 

b. To which security classes can OSF/1 be made compliant? 

c. What is the difference between compliant and certified? 

2. a. In whose context is audit information collected? 

b. How is audit information collected in an audit file? 

3. a. In what ways are ACLs more flexible than standard UNIX file protection? 

b. If DAC and MAC disagree on an access decision, how is the issue resolved? 

c. How is it determined whether a particular subject has the desired access to a particular object with respect 
to a particular access control policy? 

4. a. Explain the difference between how an authorization is implemented and how a privilege is implemented. 

b. The effective privilege set is restricted to be a subset of the potential set unioned with the base privilege 
set. Why isn't the kemel authorizations set used instead of the base privilege set? 

c. List the kemel data structures that were modified to support the OSF/1 security features. 

5. Why isn't security compliance a run-time or boot-time option? 
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Module 1 

1. page 1-2 

2. page 1-11 

3. pageMS 

4. a. page 1-21 

b. pages 1-12—1-15 

c. pages 1-16 — 1-17 

d. pages 1-18—1-19 

e. pages 1-24 — 1-25 

5. pages 1-36—1-45 

6. Mach provides the facilities to allow the efficient transfer and sharing of information across address space 
boundaries. This is particulariy usefiil for efficient communication with servers. In addition, Mach provides 
support for multiple threads of control within an address space. 

7. OSF/1 includes the logical volume manager, support for dynamic configuration, support for shared libraries 
and dynamic loading, and Bl-level security. 



Module 2 

1. a. pages 2-8 — ^2-9 

b. pages 2-8—2-9 

c. pages 2-10—2-15 

d. page 2-11 

.2. pages 2-16— 2-25 

3. a. pages 2-40— 2-47 

b. pages 2-38—2-39, 2-58—2-59 
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c. pages 2-62—2-63 

d. pages 2-50—2-51 

4. a. pages 2-68 — ^2-69 

b. pages 2-70—2-71 

c. pages 2-70—2-71 

d. pages 2-76 — ^2-77 

5. a. page 2-799 

b. page 2-81 

c. pages 2-84 — ^2-85 

6. a. pages 2-92— 2-93 

b. page 2-93 

c. pages 2-100—2-101 

d. page 2-89 

e. page 2-89 

7. a. pages 2- 104 — ^2-105 
b. pages 2-104—2-105 

8. pages 2-106— 2-107 

9. Separate ujask and proc structures are maintained in OSF/1 primarily because the separateness of tliese two 
structures is inherent in the Berkeley UNIX source code. There is no particular reason that they could not be 
merged, but there is no compelling reason to go to the effort of doing so. 

10. OSF/1 's kernel threads are cheaper than UNIX's kernel processes because kernel threads have no private 
address space associated with them. Thus the system can switch from the context of any task into the context 
of a kernel thread without changing address maps. 

1 1 . a) There are three primary reasons for nonpreemptibility in kernel mode. The first, which is not a problem in 
multiprocessor-safe operating systems such as OSF/1, is that certain data structures, if not accessed in the 
interrupt context, might have no synchronization to protea them other than the assurance that any thread in 
kernel mode will not be preempted. A second problem, which does affect OSF/1, is that a thread might be 
updating a data structure that can be accessed in the interrupt context and thus has a class of interrupts 
masked off. If this thread is preempted (because the mechanism causing preemption, e.g. clock interrupts, 
has not been masked off), then one of two things might happen, both of them bad. (1) Interrupts remain 
masked off when the system switches to the preempting thread; this is not good because the intenupt will be 
masked off much too long and may perhaps interfere with intemipt-masking done by the preempting thread. 
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(2) Interrupts become unmasked as the system enters the preempting thread's context; thus the system might 
now enter the previously masked interrupt context and access the data structure that was in the midst of 
modification by the preempted thread. This data structure, being in the middle of an update, is in an 
inconsistent state that is totally unexpected by the interrupt handler. The third reason is also a problem with 
OSF/1: if the preempted thread is holding a spin lock, the preempting thread might attempt to take this lock. 
This thread will of course spin, with no hope of taking the lock, until the preempted thread is allowed to 
execute again. 

b) The constraints on preemption points are: they must be executed only in the context of a thread, this thread 
must not be holding any locks, and no interrupts may be masked. 

Module 3 

1. a. pages 3-6 — 3-7 
b. pages 3-6—3-7 

2. a. pages 3-12— 3-13 

b. pages 3-14 — 3-17 

c. pages 3-14 — 3-17 

d. pages 3-14 — 3-17 

e. pages 3-14 — 3-17 

f. pages 3-22—3-27 

3. a. pages 3-28— 3-31 
b. pages 3-28—3-31 

4. The main problem with utilizing copy-on- write techniques to improve the implementation of UNIX system 
caUs such as write is that the best use of these techniques requires page alignment of data structures such as 
buffers. Since typical UNIX programs are not written with such alignment requirements in mind, it is 
unlikely that very many buffers would actually be properly aligned. If such alignment problems could be 
dealt with, then any UNIX system call that transfers large amounts of data could be improved. In particular, 
this means I/0-related system caUs operating on files, devices, sockets, and streams. 





Module 4 


1 


1. a. pages 4-4 — 4-5,4-49 


1 


b. page 4-5 


2. a. pages 4-10— 4-11 


1 


b. pages 4-13, 4-23, 4-73 



A-3 



Appendix — Answers to Exercises 



c. pages 4-58 — 4-59 

d. page 4-11 

3. a. pages 4-46 — 4-47 

b. pages 4-46 — 4-47 

c. pages 4-50 — 4-51 

d. page 4-37 

e. pages 4-32—4-43 

f. page 4-37 

g. pages 4-54 — 4-55 
h. pages 4-56 — 4-57 

4. a. pages 4-58— 4-101 

b. pages 4-62 — 4-65 

c. page 4-77 

d. pages 4-86— 4-97 

e. page 4-99 

5. a. pages 4-108—4-109 

b. pages 4-108—4-109 

c. pages 4-108-^-114 

d. pages 4-108—4-119 

6. If we want to enjoy the advantages of lazily evaluating the allocation of backing store, then we must deal with 
the problem that a thread's execution may fail at an arbitrary point in time. The best we can hope for is that 
the extent of the damage be limited: for example, that it be necessary to terminate only one task If this sort 
of behavior is intolerable, we may have to use a more conservative preallocation backing-store policy. 

7. The primary difficulty in replacing the vnode pager with an extemal pager is that this external pager, being 
the "pager of last resort," must never encounter page faults itself. Thus one technique might be to wire the 
extemal pager's pages into primary memory. 

Module 5 

1. page 5-9 

2. a. pages 5-16— 5-19 
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r 



b. pages 5-24 — ^5-27 

c. page 5-31 

3. a. page 5-35 

b. pages 5-40 — ^5-41 

c. pages 5-52—5-63 

4. a. pages 5-76— 5-77 
b. pages 5-80—5-81 

5. a. pages 5-88— 5-89 
b. pages 5-94—5-97 

6. a. pages 5-102— 5-103 

b. page 5-104 

c. page 5-115 

d. pages 5-122—5-123 

7. a. page 5-127 

b. page 5-131 

c. pages 5-134—5-135 

d. pages 5-138— 5-139 

e. pages 5-148—5-149 

f. pages 5-156— 5-157 

8. All three locks arc necessary. The lock on the file table entry is necessary to protect the offset stored there 
from being used prematurely by another thread. This ensures that I/O system calls executed by threads 
sharing a file table entry are atomic. The lock on buffers from the buffer cache is necessary to prevent a 
buffer from being stolen for some other purpose while one thread is using it. A simple lock is required for 
updates to the vnode because, for example, two threads concurrently reading the same file, but using different 
file table entries, might update the access time of the vnode. 

9. One might argue that, if it is known that a directory is being searched and the result of this search will be used 
very soon as part of a delete operation, the directory should be searched while holding a write lock. The 
primary reason that this is not done is that it is highly unlikely that there will be two concurrent updates to the 
same directory. Thus the optimistic approach described on page 5-75 works out very well in what is by far 
the more usual case. The fact that concurrent updates, if they occur, can be quite expensive is thus of little 
consequence. 
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Module 6 

1. a, page 6-7 
b. pages 6-8 — 6-9 

2. a. pages 6-12 — 6-13 
b. pages 6-12 — 6-13 

3. pages 6-16 — 6-17 

4. a. pages 6-36 — 6-37 

b. pages 6-40 — 6-41 

c. pages 6-46 — 6-47 

Module 7 

1. a. pages 7-6— 7-10 

b. pages 7-16 — ^7-19 

c. pages 7-20—7-23 

2. a. pages 7-31 — ^7-35 

b. pages 7-30—7-31 

c. pages 7-24 — ^7-27 

3. a. pages 7-38— 7-39 
b. pages 7-40—7-41 

4. a. pages 7-42— -7-43 
b. pages 7-48—7-49 

5. a. pages 7-50 — ^7-51 

b. pages 7-58—7-59 

c. pages 7-50—7-59 I 

6. The use of synchronization queues has no effect on the order of message processing within a streams module: 



the synchronization queues preserve the order of caUs to the module's procedures, and thus any ordering 
constraints imposed on messages by the code within a streams module is preserved. However, it is certainly 
the case that with synchronization queues there might be more messages within a streams pipeline than there 
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would be without such queues. For example, while one thread is executing within a module, other threads 
might queue more requests on this module's synchronization queue than are allowed by the high-water limit 
on the module's normal streams queue. In practice this is unlikely to be a problem: the only situation in 
which it could be a problem is if the rate at which messages are processed by a module is slower than the rate 
at which tmessages are aniving to the module. Given that streams threads execute nonpreemptively and 
without blocking, this situation is highly unlikely. 

Module 8 

1. page 8-6 

2. a. page 8-11 

b. pages 8-20—8-21 

c. page 8-5 

3. a. pages 8-26— 8-27 
b. pages 8-32— 8-33 

1. pages 8-36— 8-37 

Module 9 

1. a. pages 9-4 — ^9-5 
b. pages 9-12—9-23 

2. a. pages 9-18— 9-19,9-31 
b. page 9-19 

3. a. page 9-27 

b. page 9-22 

c. page 9-31 

d. page 9-33 

4. Only two data structures in the UFS file system depend upon the size of the entire fde system: the superblock 
and the eg summary. These of course must be modified to reflect the larger file system and the modified 
superblock must be copied to all of its alternative locations. 
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Module 10 

1. page 10-4 

2. a. pages 10-10— 10-17 
b. page 10-17 

3. a. pages 10-14— 10-19 
b. pages 10-20— 10-21 

4. a. page 10-23 

b. pages 10-23—10-24 

5. pages 10-26— 10-27 

a. pages 10-2&— 10-29 

Module 11 

1. a. page 11-7 

b. page 11-10 

c. pages 11-8—11-9 

2. a. page 11-13 

b. pages 11-14— 11-15 

3. a. pages 11-16— 11-17 

b. page 11-27 

c. page 11-27 

4. a. pages 11-28— 11-29 
b. page 11-37 

5. page 11-45 
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address space 

authorizations 

blocking lock 
bogus memory 

buffer cache 

channel 
concurrency 

cooked mode 

copy-on-write 

devices 
disposition 
exceptions 

external 
file handle 

funnel 



a set of virtual locations, such as those locations that can be referenced by a task 
or process. 

indications of whether a particular user is allowed to use a particular command or 
subsystem, perform a particular role, or gain a particular privilege. 

a lock which a thread waits for by yielding the processor. 

type of memory in a parallel architecture that does not guarantee atomicity of 
reads of aligned words. 

the collection of buffers maintained in the kernel for use in accessing files and 
block special devices. 

address of a relevant data structure that specifies an awaited event. 

multiple threads are in progress at one time; their execution might be multiplexed 
on a single processor. 

a terminal mode in which input lines can be edited, and certain charaaers cause 
signals to be sent to the process group. 

optimization using lazy evaluation in which copying is pos^ned until a task 
actually modifies a page. 

hardware. 

indicates whether or not the sleep is interruptible by a signal. 

deviations to a thread's flow of control that are caused by actions of the thread 
itself (such as addressing errors, arithmetic errors, etc.). 

outside the kernel. 

data that is used to identify a file. After a client opens a file, the server gives it a 
file handle, which the client gives to the server to speed subsequent accesses. 

a kemel data stmcture used to represent the parallel/sequential constraints of a 
particular subsystem. 



handofT scheduling a forni of thread scheduling in which one tiiread gives its processor to another. 
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idempotent 

inode 

internal 

lazy evaluation 

local port 
logical volume 

lookup cache 
memory object 

memory object manager 



message 

microkernel 

mmap 

multithreaded 

NICFREE 

package 

parallelism 

parallelization 

physical volume 



when the effect of performing an operation once is the same as perfomiing it 
multiple times, the operation is idempotent. 

the (ondisk and incore) data stmcture that describes a file (both S5 and UFS). 

inside the kemel. 

tectmique of postponing everything until the last possible moment, since if you 
put it off long enough, maybe you won*t have to do it. 

port that the message comes back through. 

an abstraction that behaves like a disk drive to file system code, but is in fact a 
collection of separate regions of real disk drives (physical volumes). 

cache of the most recent component-name-to-vnode translations. 

a '*thing" that can be mapped into a task's address space. It might be temporary 
storage (e.g., UNIX's BSS and stack), a file, or an object defined by user-provided 
servers. 

responsible for supplying initial values for a range of virtual memory and for 
backing up virtual memory when the physical memory cache becomes full. One 
may be used, for example, to map files into the address spaces of tasks, to provide 
shared memory in a distributed system, or to implement a 
transaction-management system. 

a collection of data to be sent through a port to the task that has receive rights for 
the port 

a simple, pure Mach kemel with no built-in UNIX (or other operating system) 
functionality. Such functionality would be provided by user tasks. 

a system call that is used either to map a file into a process's address space or to 
create an anonymous memory region. 

composed of a number of threads. 

number of incore free blocks. Equal to 100. 

an abstraction of a library. 

the simultaneous execution of multiple threads; requires multiple processors. 

the act of making a system parallelized. 

a real disk drive or a portion of a disk drive. 



I 
I 
I 
I 
I 
I 



G-2 



Glossary 



I 
I 

r 



I 



pmap 

port 
port set 

priority depression 

privileges 
process 

processor allocation 

processor set 

processor sharing 

raw mode 

read-ahead 
read-write lock 

remote port 

search cache 

sharing 

simple lock 

socket 

spin lock 

stream 

submap 



the data structure and code encapsulating the architecture-dependent portion of the 
virtual memory system. 

a protected queue of messages or an object reference. 

two or more ports whose message queues have been consolidated into a single 
queue by the server task. 

option to the threadjswitch system call; a calling thread's priority is "depressed" 
to the worst possible value for a given period of time, and is then restored. 

properties of a process that gain it special treatment by the operating system. 

an address space, one or more threads of control and additional information 
necessary to represent a UNIX context. 

distributing the processors of a multiprocessor among the various applications. 

mechanism for processor allocation. 

scheduling or multiplexing processors. 

the terminal mode in which incoming characters are passed immediately to user 
threads and outgoing characters are sent to the terminal with no further 
processing. 

reading the next unit of data at the same time as the current unit of data. 

a lock that can be taken as either a read lock, allowing multiple readers by no 
writers, or as a write lock, allowing a single writer and no readers. 

port for sending messages. 

a cache in the inode that contains the offset at which the last search terminated. 

what one is taught in nursery school. 

a spin lock. 

a data structure representing the end point of a communication 

a lock which a thread waits for by repeatedly testing a bit. 

the kernel analog of a sheU pipeline. 

a data structure representing a portion of the kemel address space which is 
probably managed by a single subsystem. 
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swapping unwiring or wiring the kernel stack. 

task a holder of capabilities, such as address space and communication channels. 

thread usual notion of thread of control. 

thread poo! a collection of threads used to handle events generated in the interrupt context. 

timed pause when a thread calls thread_switch with the wait option, it can be suspended for a 
fixed period of time and then automatically woken up. 

translation-lookaside buffer a hardware cache which translates virtual addresses to real addresses. 

upcall a call from a lower level of a system to a higher level (e.g. from kemel mode to 
user mode). 

virtual copy an optimized copy operation. 

virtual file system the abstraction of the file system concept: the layer of the kemel which provides 
the standard interface to the real file systems. 



vnode 



write-behind 



write-through cache 



zone 



an abstraction of a file; it contains generic information about files and refers to the 
file-system-specific information on individual files. It also refers to an array of 
entry points called vnodeops, which provides access to the various operations. 

delaying the update of a file until sometime after the write system caU has been 
completed. 

a buffer cache that requires that the data it buffers be written onto the disk before 
the system call returns. 

a collection of fixed-size blocks: a separate zone is created for each kemel data 
stmcture that is so managed. A zone is initialized with a pre-allocated free list, an 
allocation size, and a maximum size. 
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Access control, 11-17 — 11-27 

Access control list (ACL), 11-17 

Access permissions, in NFS, 5-127 

Active list, 4-55 

Address space, 2-7, 4-3, 4-7 — 4-9 
growth, 4-27 

Address space layout, 10-25 

Aliases, 6-9 

ASCn character set, 6-33 

Attach, 6-17 

Attributes, 11-23 

Audit daemon, 11-13 

Audit device driver, 11-13 

Auditing, 11-13—11-15 

Authorizations, 11-29^11-31 

B 

Backing storage allocation, 4-49 

Backup ports, 3-25—3-27 

Bad-sector remapping, 9-11 

Base priority, 2-93 

bdevsw table, 6-7, 6-13 

bdevsw_add,6-13 

Block interface, 6-5 — 6-9 

Block skip section (BSS). 4-25 

Blocked threads, 2-67 

Blocking locks, 5-51, 5-123 

Blocking threads, 2-45 — ^2-47 

Bogus memory, 5-123, 5-165 

Bootblock, 5-95 

Bootstrap port, 1-27 

bp, 6-25 

buf structures, 5-43 

Buffer cache, 5-7, 5-35—5-65 
access to, 5-41 
finding a block, 5-57—5-59 
getting a new buffer, 5-61 — 5-63 
maintenance of, 5-39 



Can queue, 6-43 — 6-45 

C^abilities, 1-21, 5-81 

cdevsw table, 6-7, 6-13 

cdevsw_add, 6-13 

Character intoface, 6-5 — 6-7 

Client-side caching, 5-133 

Clipping, 4-79 

Close, 6-17, 6-21 

Cluster pool, 8-19 

cmd, 6-27 

Collapsing objects, 4-81 

CompUance, UNIX, 1-5 

Compliant vs. certified, 11-9 

Concunency, 1-13, 5-77 

Configure, 6-17 

Configure entry point, 6-13 

Consistency 
(tevices, 6-9 
file systems, 5-45 — ^5-49 

Continue signal, 6-35 

copen, 5-29 

Copy link, 4-87 

Copy object, 3-29 

Copy-on-write, 4-59. 4-83 

Crash recovery, 9-9, 9-31 
in NFS. 5-127 

Crashes, server, 5-149—5-153 

Credentials structures, 5-137 

Cylinder group block, 5-107 

Cylinder group summary, 5-107 



Data, 6-27 

dblk,7-31 

Deadlock, 2-65 
avoiding, 2-63 



Debugging, 2-73 

Default memory object manager, 4-3 1 

Dev, 6-19— 6-29 

Device drivers, 6-3, 6-17 — 6-31, 6-43 

Device I/O, flow of C(Hitrol, 6-7 

Device module switch table, 7-45 

Device number, 6-5 

device_inuse, 2-33 

Devices, 6-5, 6-9 

Directory path searching, 5-67 — 5-83 
complications in, 5-69 

Discretionary ^cess control (DAC), 11-17 

Disk VO performance, 5-105 

Disk map, 5-89 

Disposition, 2-29 

Distributed computing environoment (DCE), 5-129 

Drain routine, 8-25 

Driver entry points, 6-17 
close, 6-21 
intOTupt, 6-29 
ioctl, 6-27 
open, 6-19 
lead/write, 6-23 
strategy, 6-25 

Dup,5-ll 

Duplicate detection, 5-163 

Dynamic configurati(»i, 6-11 — 6-15 
interrupt handle, 6-15 

Dynamic loader, 10-27—10-29 

Dynamic loading, drivers, 6-13 



Eighth-bit character sets, 6-33 

Events, 2-57 

Exception handling, in Mach, 2-75 

Exception port, 1-27, 2-77 

Exceptions, 2-69, 2-77 

Exported symbol table, 10-13, 10-17, 10-21 

Extensible loader, 1-3 

External events, 2-69 



External memcvy object managers, 1-35 



Family, 2-7 

File handles, 5-131 

File module switch table, 7-47 

File-syston-indep^dent data structures, 5-25 — ^5-27 

Hags, 4-15, 4-19, 6-19—^21, 6-27 

Flow of control, 3-29—3-31 
LVM,9-27 
open and create, 5-29 
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